Diagnosis assistance system and control method thereof

ABSTRACT

The present invention relates to a diagnosis assistance system for assisting diagnosis for a plurality of diseases based on a fundus image, the diagnosis assistance system including: a fundus image obtaining unit configured to obtain a fundus image; a first processing unit configured to, for the fundus image, obtain a first result related to a first finding of a patient using a first neural network model, a second processing unit configured to, for the fundus image, obtain a second result related to a second finding of the patient using a second neural network model, a third processing unit configured to determine, on the basis of the first result and the second result, diagnostic information on the patient, and a diagnostic information output unit configured to provide the determined diagnostic information to a user.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/359,902 filed on Mar. 20, 2019, which is a continuation ofPCT/KR2018/009809 filed on Aug. 24, 2018, which claims priority toRepublic of Korea Patent Application No. 10-2017-0108232 filed on Aug.25, 2017, which are incorporated by reference herein in their entirety.

TECHNICAL FIELD

The present invention relates to a diagnosis assistance system andcontrol method thereof, and more particularly, to a diagnosis assistancesystem and control method thereof for providing diagnosis assistanceinformation on the basis of an image.

BACKGROUND ART

The fundus examination is a diagnosis assistance material frequentlyutilized in ophthalmology since it is able to observe the abnormalitiesof the retina, optic nerve, and macula and allows the results to beconfirmed by relatively simple imaging. In recent years, the fundusexamination has been increasingly used because, through the fundusexamination, it is able to observe not only eye diseases but also adegree of blood vessel damage caused by chronic diseases such ashypertension and diabetes by a non-invasive method.

Meanwhile, due to the recent rapid development of deep learningtechnology, the development of diagnostic artificial intelligence hasbeen actively carried out in the field of medical diagnosis, especiallythe field of image-based diagnosis. Global companies such as Google andIBM have invested heavily in the development of artificial intelligencefor analyzing a variety of medical video data, including large-scaledata input through collaborations with the medical community. Somecompanies have succeeded in developing an artificial intelligencediagnostic tool that outputs superior diagnostic results.

However, in a case in which a plurality of values are desired to bepredicted from a single test data through a deep learning trained model,there has been a problem in that accuracy of prediction is reduced andprocessing speed is lowered. Accordingly, there is a need for a systemfor learning and diagnosis that enables accurate prediction of aplurality of diagnostic characteristics at a high data processing speed.

SUMMARY

One object of the present invention is to provide a neural network modeltraining method for acquiring diagnosis assistance information from afundus image.

Another object of the present invention is to provide a method oftraining a plurality of neural network models in parallel to obtain aplurality of diagnosis assistance information from a fundus image.

Still another object of the present invention is to provide a method ofpromptly acquiring a plurality of diagnosis assistance information froma fundus image by using a machine learned neural network model.

Objects to be achieved by the present invention are not limited to thosementioned above, and other unmentioned objects should be clearlyunderstood by one of ordinary skill in the art to which the presentinvention pertains from the present specification and the accompanyingdrawings.

According to an aspect of the present invention, there is provided adiagnosis assistance system for assisting diagnosis of a plurality ofdiseases based on a fundus image, the diagnosis assistance systemincluding: a fundus image obtaining unit configured to obtain a targetfundus image which is a basis for acquiring diagnosis assistanceinformation on a subject; a first processing unit configured to, for thetarget fundus image, obtain a first result related to a first finding ofthe subject using a first neural network model, wherein the first neuralnetwork model is trained on the basis of a first fundus image set; asecond processing unit configured to, for the target fundus image,obtain a second result related to a second finding of the subject usinga second neural network model, wherein the second neural network modelis trained on the basis of a second fundus image set which is at leastpartially different from the first fundus image set; a third processingunit configured to determine, on the basis of the first result and thesecond result, diagnostic information on the subject; and a diagnosticinformation output unit configured to provide the determined diagnosticinformation to a user. Here, the first finding and the second findingmay be used for diagnosing different diseases.

According to another aspect of the present invention, there is provideda training device configured to obtain a first training data setincluding a plurality of fundus images, process a fundus image includedin the first training data set, and train a first neural network modelusing the first training data set.

There is provided a control method of a training device, which isincluded in a system including a diagnostic device configured to obtaina target fundus image for obtaining diagnosis assistance information andobtain the diagnosis assistance information on the basis of the targetfundus image by using a trained first neural network model, the controlmethod including: pre-processing a first fundus image included in afirst training data set so that the first fundus image is converted to aformat facilitating the training of the first neural network model;serializing the pre-processed first fundus image; and training the firstneural network model to classify, using the serialized first fundusimage, the target fundus image to a first label or a second label.

Technical solutions of the present invention are not limited to thosementioned above, and other unmentioned technical solutions should beclearly understood by one of ordinary skill in the art to which thepresent invention pertains from the present specification and theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a diagnosis assistance system according to anembodiment of the present invention.

FIG. 2 is a block diagram for describing a training device according toan embodiment of the present invention.

FIG. 3 is a block diagram for describing the training device in moredetail according to another embodiment of the present invention.

FIG. 4 is a block diagram for describing a diagnostic device accordingto an embodiment of the present invention.

FIG. 5 is a view for describing the diagnostic device according toanother embodiment of the present invention.

FIG. 6 illustrates a diagnosis assistance system according to anembodiment of the present invention.

FIG. 7 is a block diagram for describing a client device according to anembodiment of the present invention.

FIG. 8 is a view for describing a diagnosis assistance process accordingto an embodiment of the present invention.

FIG. 9 is a view for describing a configuration of a training unitaccording to an embodiment of the present invention.

FIG. 10 is a conceptual diagram for describing an image data setaccording to an embodiment of the present invention.

FIG. 11 is a view for describing image resizing according to anembodiment of the present invention.

FIG. 12 is a view for describing expansion of an image data setaccording to an embodiment of the present invention.

FIG. 13 is a block diagram for describing a training process of a neuralnetwork model according to an embodiment of the present invention.

FIG. 14 is a block diagram for describing a training process of a neuralnetwork model according to an embodiment of the present invention.

FIG. 15 is a view for describing a control method of a training deviceaccording to an embodiment of the present invention.

FIG. 16 is a view for describing a control method of a training deviceaccording to an embodiment of the present invention.

FIG. 17 is a view for describing a control method of a training deviceaccording to an embodiment of the present invention.

FIG. 18 is a view for describing a configuration of a diagnostic unitaccording to an embodiment of the present invention.

FIG. 19 is a view for describing diagnosis target data according to anembodiment of the present invention.

FIG. 20 is a view for describing a diagnostic process according to anembodiment of the present invention.

FIG. 21 is a view for describing a parallel diagnosis assistance systemaccording to some embodiments of the present invention.

FIG. 22 is a view for describing a parallel diagnosis assistance systemaccording to some embodiments of the present invention.

FIG. 23 is a view for describing a configuration of a training deviceincluding a plurality of training units according to an embodiment ofthe present invention.

FIG. 24 is a view for describing a parallel training process accordingto an embodiment of the present invention.

FIG. 25 is a view for describing the parallel training process accordingto another embodiment of the present invention.

FIG. 26 is a block diagram for describing a diagnostic unit according toan embodiment of the present invention.

FIG. 27 is a view for describing a diagnosis assistance processaccording to an embodiment of the present invention.

FIG. 28 is a view for describing a diagnosis assistance system accordingto an embodiment of the present invention.

FIG. 29 is a view for describing a graphical user interface according toan embodiment of the present invention.

FIG. 30 is a view for describing a graphical user interface according toan embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENT

The foregoing objects, features and advantages of the present inventionwill become more apparent from the following detailed descriptionrelated to the accompanying drawings. It should be understood, however,that various modifications may be applied to the invention, and theinvention may have various embodiments. Hereinafter, specificembodiments, which are illustrated in the drawings, will be described indetail.

In the drawings, the thicknesses of layers and regions are exaggeratedfor clarity. When it is indicated that an element or layer is “on” or“above” another element or layer, this includes a case in which anotherlayer or element is interposed therebetween as well as a case in whichthe element or layer is directly above the other element or layer. Inprinciple, like reference numerals designate like elements throughoutthe specification. In the following description, like reference numeralsare used to designate elements which have the same function within thesame idea illustrated in the drawings of each embodiment.

When detailed description of known functions or configurations relatedto the present invention is deemed to unnecessarily blur the gist of theinvention, the detailed description thereof will be omitted. Also,numerals (e.g., first, second, etc.) used in the description herein aremerely identifiers for distinguishing one element from another element.

In addition, the terms “module” and “unit” used to refer to elements inthe following description are given or used in combination only inconsideration of ease of writing the specification, and the termsthemselves do not have distinct meanings or roles.

A method according to an embodiment may be implemented in the form of aprogram command that can be executed through various computer means andmay be recorded in a computer-readable medium. The computer-readablemedium may include program commands, data files, data structures, andthe like alone or in combination. The program commands recorded in themedium may be those specially designed and configured for the embodimentor those known to those skilled in the art of computer software andusable. Examples of the computer-readable recording medium includemagnetic media such as a hard disk, a floppy disk, and a magnetic tape,optical media such as compact disk-read only memory (CD-ROM), and adigital versatile disk (DVD), magneto-optical media such as a flopticaldisk, and hardware devices such as a read only memory (ROM), a randomaccess memory (RAM), and a flash memory specially configured to storeand execute a program command. Examples of the program command includehigh-level language codes that may be executed by a computer using aninterpreter or the like as well as machine language codes generated by acompiler. The above-mentioned hardware device may be configured tooperate as one or more software modules to execute operations accordingto an embodiment, and vice versa.

1. System and Process for Diagnosis Assistance 1.1 Purpose andDefinition

Hereinafter, a system and method for diagnosis assistance for assistingin determination of the presence of a disease or illness on the basis ofa fundus image or the presence of an abnormality which is a basis of thedetermination will be described. Particularly, a system or method fordiagnosis assistance in which a neural network model for diagnosing adisease is constructed using a deep learning technique and detection ofthe presence of a disease or abnormal findings is assisted using theconstructed model will be described.

According to an embodiment of the present invention, a system or methodfor diagnosis assistance in which diagnostic information related to thepresence of a disease, findings information used in diagnosis of thepresence of a disease, or the like are obtained on the basis of a fundusimage and diagnosis is assisted using the obtained information may beprovided.

According to an embodiment of the present invention, a system or methodfor diagnosis assistance in which diagnosis of an eye disease isassisted on the basis of a fundus image may be provided. For example, asystem or method for diagnosis assistance in which diagnosis is assistedby obtaining diagnostic information related to the presence of glaucoma,cataract, macular degeneration, retinopathy of prematurity of a testeemay be provided.

According to another embodiment of the present invention, a system ormethod for diagnosis assistance in which diagnosis of a disease otherthan an eye disease (for example, a systemic disease or a chronicdisease) is assisted may be provided. For example, a system or methodfor diagnosis assistance in which diagnosis is assisted by obtainingdiagnostic information on a systemic disease such as hypertension,diabetes, Alzheimer's, cytomegalovirus, stroke, heart disease, andarteriosclerosis may be provided.

According to still another embodiment of the present invention, a systemor method for diagnosis assistance for detecting abnormal fundusfindings that may be used in diagnosis of an eye disease or otherdiseases may be provided. For example, a system or method for diagnosisassistance for obtaining findings information such as abnormal color ofthe entire fundus, opacity of crystalline lens, abnormal cup-to-disc(C/D) ratio, macular abnormalities (e.g., macular hole), an abnormaldiameter or course of a blood vessel, an abnormal diameter of theretinal artery, retinal hemorrhage, generation of exudate, and drusenmay be provided.

In the specification, diagnosis assistance information may be understoodas encompassing diagnostic information according to determination of thepresence of a disease, findings information which is a basis of thedetermination, or the like.

1.2 Configuration of Diagnosis Assistance System

According to an embodiment of the present invention, a diagnosisassistance system may be provided.

FIG. 1 illustrates a diagnosis assistance system 10 according to anembodiment of the present invention. Referring to FIG. 1, the diagnosisassistance system 10 may include a training device 1000 configured totrain a diagnostic model, a diagnostic device 2000 configured to performdiagnosis using the diagnostic model, and a client device 3000configured to obtain a diagnosis request. The diagnosis assistancesystem 10 may include a plurality of training devices, a plurality ofdiagnostic devices, or a plurality of client devices.

The training device 1000 may include a training unit 100. The trainingunit 100 may perform training of a neural network model. For example,the training unit 100 may obtain a fundus image data set and performtraining of a neural network model that detects a disease or abnormalfindings from a fundus image.

The diagnostic device 2000 may include a diagnostic unit 200. Thediagnostic unit 200 may perform diagnosis of a disease or obtainassistance information used for the diagnosis by using a neural networkmodel. For example, the diagnostic unit 200 may obtain diagnosisassistance information by using a diagnostic model trained by thetraining unit.

The client device 3000 may include an imaging unit 300. The imaging unit300 may capture a fundus image. The client device may be an ophthalmicfundus imaging device. Alternatively, the client device 3000 may be ahandheld device such as a smartphone or a tablet personal computer (PC).

In the diagnosis assistance system 10 according to the presentembodiment, the training device 1000 may obtain a data set and train aneural network model to determine a neural network model to be used indiagnosis assistance, the diagnostic device may obtain diagnosisassistance information according to a diagnosis target image by usingthe determined neural network model when an information request isobtained from the client device, and the client device may request thediagnostic device for information and obtain diagnosis assistanceinformation transmitted in response to the request.

A diagnosis assistance system according to another embodiment mayinclude a diagnostic device configured to train a diagnostic model andperform diagnosis using the same and may include a client device. Adiagnosis assistance system according to still another embodiment mayinclude a diagnostic device configured to train a diagnostic model,obtain a diagnosis request, and perform diagnosis. A diagnosisassistance system according to yet another embodiment may include atraining device configured to train a diagnostic model and a diagnosticdevice configured to obtain a diagnosis request and perform diagnosis.

The diagnosis assistance system disclosed herein is not limited to theabove-described embodiments and may be implemented in any form includinga training unit configured to train a model, a diagnostic unitconfigured to obtain diagnosis assistance information according to thetrained image, and an imaging unit configured to obtain a diagnosistarget image.

Hereinafter, some embodiments of each device constituting the systemwill be described.

1.2.1 Training Device

A training device according to an embodiment of the present inventionmay train a neural network model that assists diagnosis.

FIG. 2 is a block diagram for describing a training device 1000according to an embodiment of the present invention. Referring to FIG.2, the training device 1000 may include a control unit 1200 and a memoryunit 1100.

The training device 1000 may include the control unit 1200. The controlunit 1200 may control operation of the training device 1000.

The control unit 1200 may include one or more of a central processingunit (CPU), a random access memory (RAM), a graphic processing unit(GPU), one or more microprocessors, and an electronic component capableof processing input data according to predetermined logic.

The control unit 1200 may read a system program and various processingprograms stored in the memory unit 1100. For example, the control unit1200 may develop a data processing process for performing diagnosisassistance which will be described below, a diagnostic process, and thelike in a RAM and perform various processes according to a developedprogram. The control unit 1200 may perform training of a neural networkmodel which will be described below.

The training device 1000 may include the memory unit 1100. The memoryunit 1100 may store data required for training and a training model.

The memory unit 1100 may be implemented using a nonvolatilesemiconductor memory, a hard disk, a flash memory, a RAM, a ROM, anelectrically erasable programmable ROM (EEPROM), or other tangiblenonvolatile recording media.

The memory unit 1100 may store various processing programs, parametersfor processing programs, result data of such processing, or the like.For example, the memory unit 1100 may store a data processing processprogram for performing diagnosis assistance which will be describedbelow, a diagnostic process program, parameters for executing eachprogram, data obtained according to execution of such programs (forexample, processed data or diagnosis result values), and the like.

The training device 1000 may include a separate training unit (ortraining module). The training unit may train a neural network model.The training will be described in more detail below in Section “2.Training process.”

The training unit may be included in the above-described control unit1200. The training unit may be stored in the above-described memory unit1100. The training unit may be implemented by partial configurations ofthe above-described control unit 1200 and memory unit 1100. For example,the training unit may be stored in the memory unit 1100 and driven bythe control unit 1200.

The training device 1000 may further include a communication unit 1300.The communication unit 1300 may communicate with an external device. Forexample, the communication unit 1300 may communicate with a diagnosticdevice, a server device, or a client device which will be describedbelow. The communication unit 1300 may perform wired or wirelesscommunication. The communication unit 1300 may perform bidirectional orunidirectional communication.

FIG. 3 is a block diagram for describing the training device 1000 inmore detail according to another embodiment of the present invention.Referring to FIG. 3, the training device 1000 may include a processor1050, a volatile memory 1030, a nonvolatile memory 1010, a mass storagedevice 1070, and a communication interface 1090.

The processor 1050 of the training device 1000 may include a dataprocessing module 1051 and a training module 1053. The processor 1050may process a data set stored in the mass storage device or nonvolatilememory through the data processing module 1051. The processor 1050 maytrain a diagnosis assistance neural network model through the trainingmodule 1053. The processor 1050 may include a local memory. Thecommunication interface 1090 may be connected to a network 1110.

However, the training device 1000 illustrated in FIG. 3 is merely anexample, and the configuration of the training device 1000 according tothe present invention is not limited thereto. Particularly, the dataprocessing module or training module may be provided at locationsdifferent from those illustrated in FIG. 3.

1.2.2 Diagnostic Device

A diagnostic device may obtain diagnosis assistance information using aneural network model.

FIG. 4 is a block diagram for describing a diagnostic device 2000according to an embodiment of the present invention. Referring to FIG.4, the diagnostic device 2000 may include a control unit 2200 and amemory unit 2100.

The control unit 2200 may generate diagnosis assistance informationusing a diagnosis assistance neural network model. The control unit 2200may obtain diagnostic data for diagnosis (for example, fundus data of atestee) and obtain diagnosis assistance information predicted by thediagnostic data using a trained diagnosis assistance neural networkmodel.

The memory unit 2100 may store a trained diagnosis assistance neuralnetwork model. The memory unit 2100 may store parameters, variables, andthe like of a diagnosis assistance neural network model.

The diagnostic device 2000 may further include a communication unit2300. The communication unit 2300 may communicate with a training deviceand/or a client device. For example, the diagnostic device 2000 may beprovided in the form of a server that communicates with a client device.This will be described in more detail below.

FIG. 5 is a view for describing the diagnostic device 2000 according toanother embodiment of the present invention. Referring to FIG. 5, thediagnostic device 2000 according to an embodiment of the presentinvention may include a processor 2050, a volatile memory 2030, anonvolatile memory 2010, a mass storage device 2070, and a communicationinterface 2090.

The processor 2050 of the diagnostic device may include a dataprocessing module 2051 and a diagnostic module 2053. The processor 2050may process diagnostic data through the data processing module 2051 andobtain diagnosis assistance information according to the diagnostic datathrough the diagnostic module 2053.

1.2.3 Server Device

According to an embodiment of the present invention, a diagnosisassistance system may include a server device. The diagnosis assistancesystem according to an embodiment of the present invention may alsoinclude a plurality of server devices.

The server device may store and/or drive a neural network model. Theserver device may store weights constituting a trained neural networkmodel. The server device may collect or store data used in diagnosisassistance.

The server device may output a result of a diagnosis assistance processusing a neural network model to a client device. The server device mayobtain feedback from the client device. The server device may operatesimilar to the above-described diagnostic device.

FIG. 6 illustrates a diagnosis assistance system 20 according to anembodiment of the present invention. Referring to FIG. 6, the diagnosisassistance system 20 according to an embodiment of the present inventionmay include a diagnostic server 4000, a training device, and a clientdevice.

The diagnostic server 4000, i.e., server device, may communicate with aplurality of training devices or a plurality of diagnostic devices.Referring to FIG. 6, the diagnostic server 4000 may communicate with afirst training device 1000 a and a second training device 1000 b.Referring to FIG. 6, the diagnostic server 4000 may communicate with afirst client device 3000 a and a second client device 3000 b.

For example, the diagnostic server 4000 may communicate with the firsttraining device 1000 a configured to train a first diagnosis assistanceneural network model that obtains a first diagnosis assistanceinformation and the second training device 1000 b configured to train asecond diagnosis assistance neural network model that obtains a seconddiagnosis assistance information.

The diagnostic server 4000 may store the first diagnosis assistanceneural network model that obtains the first diagnosis assistanceinformation and the second diagnosis assistance neural network modelthat obtains the second diagnosis assistance information, obtaindiagnosis assistance information in response to a request for obtainingdiagnosis assistance information from the first client device 3000 a orthe second client device 3000 b, and transmit the obtained diagnosisassistance information to the first client device 3000 a or the secondclient device 3000 b.

Alternatively, the diagnostic server 4000 may communicate with the firstclient device 3000 a that requests for the first diagnosis assistanceinformation and the second client device 3000 b that requests for thesecond diagnosis assistance information.

1.2.4 Client Device

A client device may request a diagnostic device or a server device fordiagnosis assistance information. The client device may obtain datarequired for diagnosis and transmit the obtained data to the diagnosticdevice.

The client device may include a data obtaining unit. The data obtainingunit may obtain data required for diagnosis assistance. The dataobtaining unit may be an imaging unit configured to obtain an image usedin a diagnosis assistance model.

FIG. 7 is a block diagram for describing the client device 3000according to an embodiment of the present invention. Referring to FIG.7, the client device 3000 according to an embodiment of the presentinvention may include an imaging unit 3100, a control unit 3200, and acommunication unit 3300.

The imaging unit 3100 may obtain image or video data. The imaging unit3100 may obtain a fundus image. However, in the client device 3000, theimaging unit 3100 may also be substituted with another form of dataobtaining unit.

The communication unit 3300 may communicate with an external device,e.g., a diagnostic device or a server device. The communication unit3300 may perform wired or wireless communication.

The control unit 3200 may control the imaging unit 3100 to obtain imagesor data. The control unit 3200 may control the imaging unit 3100 toobtain a fundus image. The control unit 3200 may transmit the obtainedfundus image to the diagnostic device. The control unit may transmit animage obtained through the imaging unit 3100 to the server devicethrough the communication unit 3300 and obtain diagnosis assistanceinformation generated on the basis of the obtained image.

Although not illustrated, the client device may further include anoutput unit. The output unit may include a display configured to outputa video or an image or may include a speaker configured to output sound.The output unit may output video or image data obtained by the imagingunit. The output unit may output diagnosis assistance informationobtained from the diagnostic device.

Although not illustrated, the client device may further include an inputunit. The input unit may obtain a user input. For example, the inputunit may obtain a user input that requests for diagnosis assistanceinformation. The input unit may obtain information on a user whoevaluates diagnosis assistance information obtained from the diagnosticdevice.

In addition, although not illustrated, the client device may furtherinclude a memory unit. The memory unit may store an image obtained bythe imaging unit.

1.3 Outline of Diagnosis Assistance Process

A diagnosis assistance process may be performed by a diagnosisassistance system or a diagnosis assistance device disclosed herein. Thediagnosis assistance process may be taken into consideration by beingmainly divided into a training process for training a diagnosisassistance model used in diagnosis assistance and a diagnostic processusing the diagnosis assistance model.

FIG. 8 is a view for describing a diagnosis assistance process accordingto an embodiment of the present invention. Referring to FIG. 8, thediagnosis assistance process according to an embodiment of the presentinvention may include a training process including obtaining andprocessing data (S110), training a neural network model (S130), andobtaining variables of the trained neural network model (S150) and adiagnosis assistance process including obtaining diagnosis target data(S210), using a neural network model trained on the basis of thediagnosis target data (S230), and obtaining diagnosis assistanceinformation using the trained neural network model (S250).

More specifically, the training process may include a data processingprocess in which input training image data is processed to a state inwhich the data may be used for model training and a training process inwhich a model is trained using the processed data. The training processmay be performed by the above-described training device.

The diagnostic process may include a data processing process in whichinput examination target image data is processed to a state in whichdiagnosis using a neural network model may be performed and a diagnosticprocess in which diagnosis is performed using the processed data. Thediagnostic process may be performed by the above-described diagnosticdevice or server device.

Hereinafter, each process will be described.

2. Training Process

According to an embodiment of the present invention, a process fortraining a neural network model may be provided. As a specific example,a process for training a neural network model that performs or assistsdiagnosis on the basis of a fundus image may be disclosed.

The training process which will be described below may be performed bythe above-described training device.

2.1 Training Unit

According to an embodiment of the present invention, a training processmay be performed by a training unit. The training unit may be providedin the above-described training device.

FIG. 9 is a view for describing a configuration of a training unit 100according to an embodiment of the present invention. Referring to FIG.9, the training unit 100 may include a data processing module 110, aqueue module 130, a training module 150, and a training result obtainingmodule 170. As will be described below, the modules may performindividual steps of a data processing process and a training process.However, not all of the elements described with reference to FIG. 9 andfunctions performed by the elements are essential, and some elements maybe added or omitted according to a form of training.

2.2 Data Processing Process 2.2.1 Obtaining Image Data

According to an embodiment of the present invention, a data set may beobtained. According to an embodiment of the present invention, a dataprocessing module may obtain a data set.

The data set may be an image data set. Specifically, the data set may bea fundus image data set. The fundus image data set may be obtained usinga general non-mydriatic fundus camera or the like. A fundus image may bea panorama image. The fundus image may be a red-free image. The fundusimage may be an infrared image. The fundus image may be anautofluorescence image. The image data may be obtained in any one formatamong JPG, PNG, DCM (DICOM), BMP, GIF, and TIFF.

The data set may include a training data set. The data set may include atest data set. The data set may include a validation data set. In otherwords, the data set may be assigned as at least one of a training dataset, a test data set, and a validation data set.

The data set may be determined in consideration of diagnosis assistanceinformation that is desired to be obtained using a neural network modeltrained through the corresponding data set. For example, when it isdesired to train a neural network model that obtains diagnosisassistance information related to cataract, an infrared fundus imagedata set may be determined as a data set to be obtained. Alternatively,when it is desired to train a neural network model that obtainsdiagnosis assistance information related to macular degeneration, anobtained data set may be an autofluorescence fundus image data set.

Individual data included in a data set may include a label. There may bea plurality of labels. In other words, individual data included in adata set may be labeled in relation to at least one feature. Forexample, a data set may be a fundus image data set including a pluralityof fundus image data, and each fundus image data may include a labelrelated to diagnostic information (for example, the presence of aspecific disease) and/or a label related to findings information (forexample, whether a specific site is abnormal) according to thecorresponding image.

As another example, a data set may be a fundus image data set, and eachfundus image data may include a label related to peripheral informationon the corresponding image. For example, each fundus image data mayinclude a label related to peripheral information including lefteye/right eye information on whether the corresponding fundus image isan image of the left eye or an image of the right eye, genderinformation on whether the corresponding fundus image is a fundus imageof a female or a fundus image of a male, age information on the age of atestee to which the corresponding fundus image belongs, and the like.

FIG. 10 is a conceptual diagram for describing an image data set DSaccording to an embodiment of the present invention. Referring to FIG.10, the image data set DS according to an embodiment of the presentinvention may include a plurality of image data ID. Each image data IDmay include an image I and a label L assigned to the image. Referring toFIG. 10, the image data set DS may include a first image data ID1 and asecond image data ID2. The first image data ID1 may include a firstimage I1 and a first label L1 corresponding to the first image.

Although the case in which a single image data includes a single labelhas been described above with reference to FIG. 10, a single image datamay include a plurality of labels as described above.

2.2.2 Image Resizing

According to an embodiment of the present invention, the size of anobtained piece of image data may be adjusted. That is, images may beresized. According to an embodiment of the present invention, imageresizing may be performed by the data processing module of theabove-described training unit.

The size or aspect ratio of an image may be adjusted. Sizes of aplurality of obtained images may be adjusted so that the images have acertain size. Alternatively, the sizes of the images may be adjusted sothat the images have a certain aspect ratio. Resizing an image mayinclude applying an image conversion filter to an image.

When the sizes or capacities of obtained individual images areexcessively large or small, the size or volume of an image may beadjusted to convert the image to an appropriate size. Alternatively,when the sizes or capacities of individual images vary, the sizes orcapacities may be made uniform through resizing.

According to an embodiment, a volume of an image may be adjusted. Forexample, when a volume of an image exceeds an appropriate range, theimage may be reduced through down-sampling. Alternatively, when a volumeof an image is below an appropriate range, the image may be enlargedthrough up-samplling or interpolating.

According to another embodiment, an image may be cut or pixels may beadded to an obtained image to adjust the size or aspect ratio of theimage. For example, when a portion unnecessary for training is includedin an image, a portion of the image may be cropped to remove theunnecessary portion. Alternatively, when a portion of the image is cutaway and a set aspect ratio is not met, a column or row may be added tothe image to adjust the aspect ratio of the image. In other words, amargin or padding may be added to the image to adjust the aspect ratio.

According to still another embodiment, the volume and the size or aspectratio of the image may be adjusted together. For example, when a volumeof an image is large, the image may be down-sampled to reduce the volumeof the image, and an unnecessary portion included in the reduced imagemay be cropped to convert the image to appropriate image data.

According to another embodiment of the present invention, an orientationof image data may be changed.

As a specific example, when a fundus image data set is used as a dataset, the volume or size of each fundus image may be adjusted. Croppingmay be performed to remove a margin portion excluding a fundus portionof a fundus image, or padding may be performed to supplement a cut-awayportion of a fundus image and adjust an aspect ratio thereof.

FIG. 11 is a view for describing image resizing according to anembodiment of the present invention. Referring to FIG. 11, an obtainedfundus image may be resized by an image resizing process according to anembodiment of the present invention.

Specifically, an original fundus image (a) may be cropped as shown in(b) so that a margin portion unnecessary for obtaining diagnosticinformation is removed or the size thereof may be reduced as shown in(c) for enhancing the training efficiency.

2.2.3 Image Pre-Processing

According to an embodiment of the present invention, imagepre-processing may be performed. When an input image is used as it is intraining, an overfitting phenomenon may occur as a result of a trainingfor unnecessary characteristics, and the training efficiency may also bedegraded.

To prevent this, image data may be appropriately pre-processed to servea purpose of training, thereby improving the efficiency and performanceof training. For example, pre-processing of a fundus image may beperformed to facilitate detection of abnormal symptoms of an eyedisease, or pre-processing of a fundus image may be performed so thatchanges in retinal vessels or blood flow are emphasized.

Image pre-processing may be performed by the data processing module ofthe above-described training unit. The data processing module may obtaina resized image and perform pre-processing required for training.

Image pre-processing may be performed on the above-mentioned resizedimage. However, content of the invention disclosed herein is not limitedthereto, and image pre-processing may also be performed without theresizing process. Pre-processing an image may include applying apre-processing filter to the image.

According to an embodiment, a blur filter may be applied to an image. AGaussian filter may be applied to an image. A Gaussian blur filter mayalso be applied to an image. Alternatively, a deblur filter whichsharpens an image may be applied to the image.

According to another embodiment, a filter that adjusts or modulatescolor of an image may be applied. For example, a filter that changesvalues of some components of RGB values constituting an image orbinarizes the image may be applied.

According to still another embodiment, a filter that causes a specificelement in an image to be emphasized may be applied to the image. Forexample, pre-processing that causes a blood vessel element to beemphasized from each image may be performed on fundus image data. Inthis case, the pre-processing that causes a blood vessel element to beemphasized may include applying one or more filters sequentially or incombination.

According to an embodiment of the present invention, imagepre-processing may be performed in consideration of a characteristic ofdiagnosis assistance information that is desired to be obtained. Forexample, when it is desired to obtain diagnosis assistance informationrelated to findings such as retinal hemorrhage, drusen, microaneurysms,and exudates, pre-processing that converts an obtained fundus image intoa red-free fundus image may be performed.

2.2.4 Image Augmentation

According to an embodiment of the present invention, an image may beaugmented or expanded. Image augmentation may be performed by the dataprocessing module of the above-described training unit.

Augmented images may be used for improving performance of training aneural network model. For example, when an amount of data for training aneural network model is insufficient, existing training image data maybe modulated to increase the number of data for training, and modulated(or modified) images may be used together with an original image,thereby increasing the number of training image data. Accordingly,overfitting may be suppressed, layers of a model may be formed deeper,and accuracy of prediction may be improved.

For example, expansion of image data may be performed by reversing theleft and right of an image, cutting(cropping) a part of the image,correcting a color value of the image, or adding artificial noise to theimage. As a specific example, cutting a part of the image may beperformed by cutting a partial region of an element constituting animage or randomly cutting partial regions. In addition, image data maybe expanded by reversing the left and right of the image data, reversingthe top and bottom of the image data, rotating the image data, resizingthe image data to a certain ratio, cropping the image data, padding theimage data, adjusting color of the image data, or adjusting brightnessof the image data.

For example, the above-described augmentation or expansion of image datamay be generally applied to a training data set. However, theaugmentation or expansion of image data may also be applied to otherdata sets, for example, a test data set, i.e., a data set for testing amodel on which training using training data and validation usingvalidation data have been completed.

As a specific example, when a fundus image data set is used as a dataset, an augmented fundus image data set may be obtained by randomlyapplying one or more processes of reversing an image, cutting an image,adding noise to an image, and changing color of an image to increase thenumber of data.

FIG. 12 is a view for describing expansion of an image data setaccording to an embodiment of the present invention. Referring to FIG.12, an image according to embodiments of the present invention may bedeformed to improve prediction accuracy of a neural network model.

Specifically, referring to FIG. 12, partial regions may be dropped outfrom an image according to embodiments of the present invention as shownin (a), the left and right of the image may be reversed as shown in (b),a central point of the image may be moved as shown in (c) and (d), andcolor of partial regions of the image may be modulated as shown in (e).

2.2.5 Image Serialization

According to an embodiment of the present invention, image data may beserialized. An image may be serialized by the data processing module ofthe above-described training unit. A serializing module may serializepre-processed image data and transmit the serialized image data to aqueue module.

When image data is used as it is in training, since the image data hasan image file format such as JPG, PNG, and DCM, decoding is necessary.However, when training is performed through decoding every time,performance of training a model may be degraded. Accordingly, trainingmay be performed using an serialized image instead of using the imagefile as it is in training. Therefore, image data may be serialized toimprove the performance and speed of training. The image data beingserialized may be image data to which one or more steps of theabove-described image resizing and image pre-processing are applied ormay be image data on which neither the image resizing nor the imagepre-processing has been processed.

Each piece of image data included in an image data set may be convertedto a string format. Image data may be converted to a binarized dataformat. Particularly, image data may be converted to a data formatsuitable for use in training a neural network model. For example, imagedata may be converted to the TFRecord format for use in training aneural network model using Tensorflow.

As a specific example, when a fundus image set is used as a data set,the obtained fundus image set may be converted to the TFRecord formatand used in training a neural network model.

2.2.6 Queue

A queue may be used for solving a data bottleneck phenomenon. The queuemodule of the above-described training unit may store image data in aqueue and transmit the image data to a training module.

Particularly, when a training process is performed by using a CPU and aGPU together, a bottleneck phenomenon between the CPU and the GPU may beminimized, access to a database may be facilitated, and the memory usageefficiency may be enhanced by using a queue.

A queue may store data used in training a neural network model. Thequeue may store image data. The image data stored in the queue may beimage data on which at least one of the above-described data processingprocesses (that is, resizing, pre-processing, and augmentation) areprocessed or may be image data that is unchanged after being obtained.

A queue may store image data, preferably, serialized image data asdescribed above. The queue may store image data and supply the imagedata to a neural network model. The queue may transfer image data inbatch size to a neural network model.

A queue may provide image data. The queue may provide data to a trainingmodule which will be described below. As data is extracted from thetraining module, the number of data accumulated in the queue may bedecreased.

When the number of data stored in the queue is decreased to a referencenumber or lower as training of a neural network model is performed, thequeue may request for supplementation of data. The queue may request forsupplementation of a specific type of data. When the queue requests thetraining unit for supplementation of data, the training unit maysupplement the queue with data.

A queue may be provided in a system memory of the training device. Forexample, the queue may be formed in a RAM of a CPU. In this case, thesize, i.e., volume, of the queue may be set according to the capacity ofthe RAM of the CPU. A first-in-first-out (FIFO) queue, a primary queue,or a random queue may be used as the queue.

2.3 Training Process

According to an embodiment of the present invention, a training processof a neural network model may be disclosed.

According to an embodiment of the present invention, training of aneural network model may be performed by the above-described trainingdevice. A training process may be performed by the control unit of thetraining device. A training process may be performed by the trainingmodule of the above-described training unit.

FIG. 13 is a block diagram for describing a training process of a neuralnetwork model according to an embodiment of the present invention.Referring to FIG. 13, a training process of a neural network modelaccording to an embodiment of the present invention may be performed byobtaining data (S1010), training a neural network model (S1030),validating the trained model (S1050), and obtaining variables of thetrained model (S1070).

Hereinafter, some embodiments of a training process of a neural networkmodel will be described with reference to FIG. 13.

2.3.1 Data Input

A data set for training a diagnosis assistance neural network model maybe obtained.

Obtained data may be an image data set processed by the above-describeddata processing process. For example, a data set may include fundusimage data which is adjusted in size, has a pre-processing filterapplied thereto, is augmented and then serialized.

In training a neural network model, a training data set may be obtainedand used. In validating the neural network model, a validation data setmay be obtained and used. In testing the neural network model, a testdata set may be obtained and used. Each data set may include fundusimages and labels.

A data set may be obtained from a queue. The data set may be obtained inbatches from the queue. For example, when sixty data sets are designatedas the size of a batch, sixty data sets may be extracted at a time fromthe queue. The size of a batch may be limited by the capacity of a RAMof a GPU.

A data set may be randomly obtained from a queue by the training module.Data sets may also be obtained in order of being accumulated in thequeue.

The training module may extract a data set by designating aconfiguration of a data set to be obtained from the queue. For example,the training module may extract fundus image data having a left eyelabel of a specific patient and fundus image data having a right eyelabel of the specific patient to be used together in training.

The training module may obtain a data set having a specific label fromthe queue. For example, the training module may obtain fundus image datain which a diagnostic information label is abnormal label from thequeue. The training module may obtain a data set from the queue bydesignating a ratio between numbers of data according to certain labels.For example, the training module may obtain a fundus image data set fromthe queue so that the number of fundus image data in which a diagnosticinformation label is abnormal and the number of fundus image data inwhich the diagnostic information label is normal has a 1:1 ratio.

2.3.2 Model Design

A neural network model may be a diagnosis assistance model that outputsdiagnosis assistance information on the basis of image data. A structureof a diagnosis assistance neural network model for obtaining diagnosisassistance information may have a predetermined form. The neural networkmodel may include a plurality of layers.

A neural network model may be implemented in the form of a classifierthat generates diagnosis assistance information. The classifier mayperform binary classification or multiclass classification. For example,a neural network model may be a binary classification model thatclassifies input data as a normal or abnormal class in relation totarget diagnosis assistance information such as a specific disease orabnormal symptoms. Alternatively, a neural network model may be amulticlass classification model that classifies input data into aplurality of classes in relation to a specific characteristic (forexample, a degree of disease progression). Alternatively, a neuralnetwork model may be implemented as a regression model that outputsspecific values related to a specific disease.

A neural network model may include a convolutional neural network (CNN).As a CNN structure, at least one of AlexNet, LENET, NIN, VGGNet, ResNet,WideResnet, GoogleNet, FractaNet, DenseNet, FitNet, RitResNet,HighwayNet, MobileNet, and DeeplySupervisedNet may be used. The neuralnetwork model may be implemented using a plurality of CNN structures.

For example, a neural network model may be implemented to include aplurality of VGGNet blocks. As a more specific example, a neural networkmodel may be provided by coupling between a first structure in which a3×3 CNN layer having 64 filters, a batch normalization (BN) layer, and aReLu layer are sequentially coupled and a second block in which a 3×3CNN layer having 128 filters, a ReLu layer, and a BN layer aresequentially coupled.

A neural network model may include a max pooling layer subsequent toeach CNN block and include a global average pooling (GAP) layer, a fullyconnected (FC) layer, and an activation layer (for example, sigmoid,softmax, and the like) at an end.

2.3.3 Model Training

A neural network model may be trained using a training data set.

A neural network model may be trained using a labeled data set. However,a training process of a diagnosis assistance neural network modeldescribed herein is not limited thereto, and a neural network model mayalso be trained in an unsupervised form using unlabeled data.

Training of a neural network model may be performed by obtaining aresult value using a neural network model to which arbitrary weights areassigned on the basis of training image data, comparing the obtainedresult value with a label value of the training data, and performingbackpropagation according to an error therebetween to optimize theweights. Also, training of a neural network model may be affected by aresult of validating the model, a result of testing the model, and/orfeedback on the model received from the diagnosis step.

The above-described training of a neural network model may be performedusing Tensorflow. However, the present invention is not limited thereto,and a framework such as Theano, Keras, Caffe, Torch, and MicrosoftCognitive Toolkit (CNTK) may also be used in training a neural networkmodel.

2.3.4 Model Validation

A neural network model may be validated using a validation data set.Validation of a neural network model may be performed by obtaining aresult value related to a validation data set from a neural networkmodel which has been trained and comparing the result value with a labelof the validation data set. The validation may be performed by measuringaccuracy of the result value. Parameters of a neural network model (forexample, weights and/or bias) or hyperparameters (for example, learningrate) of the neural network model may be adjusted according to avalidation result.

For example, the training device according to an embodiment of thepresent invention may train a neural network model that predictsdiagnosis assistance information on the basis of a fundus image andcompare diagnosis assistance information on a validated fundus image ofthe trained model with a validation label corresponding to the validatedfundus image to perform validation of the diagnosis assistance neuralnetwork model.

In validation of a neural network model, an external data set, that is,a data set having a distinguished factor not included in a training dataset, may be used. For example, the external data set may be a data setin which factors such as race, environment, age, and gender aredistinguished from the training data set.

2.3.5 Model Test

A neural network model may be tested using a test data set.

Although not illustrated in FIG. 13, according to the training processaccording to an embodiment of the present invention, a neural networkmodel may be tested using a test data set which is differentiated from atraining data set and a validation data set. Parameters of a neuralnetwork model (for example, weights and/or bias) or hyperparameters (forexample, learning rate) of the neural network model may be adjustedaccording to a test result.

For example, the training device according to an embodiment of thepresent invention may obtain a result value which has test fundus imagedata, which has not been used in the training and validation, as inputfrom the neural network model which has been trained to predictdiagnosis assistance information on the basis of a fundus image and mayperform testing of the diagnosis assistance neural network model whichhas been trained and validated.

In testing of the neural network model, an external data set, that is, adata set having a factor distinguished from the training data set and/orvalidation data set, may be used.

2.3.6 Output of Result

As a result of training a neural network model, optimized parametervalues of the model may be obtained. As training of the model using atest data set as described above is repeatedly performed, moreappropriate parameter (variable) values may be obtained. When thetraining is sufficiently performed, optimized values of weights and/orbias may be obtained.

According to an embodiment of the present invention, a trained neuralnetwork model and/or parameters or variables of the trained neuralnetwork model may be stored in the training device and/or diagnosticdevice (or server). The trained neural network model may be used inpredicting diagnosis assistance information by the diagnostic deviceand/or client device. Also, the parameters or variables of the trainedneural network model may be updated by feedback obtained from thediagnostic device or client device.

2.3.7 Model Ensemble

According to an embodiment of the present invention, in a process oftraining a single diagnosis assistance neural network model, a pluralityof sub-models may be simultaneously trained. The plurality of sub-modelsmay have different layer structures.

In this case, the diagnosis assistance neural network model according toan embodiment of the present invention may be implemented by combining aplurality of sub-neural network models. In other words, training of aneural network model may be performed using an ensemble technique inwhich a plurality of sub-neural network models are combined.

When a diagnosis assistance neural network model is configured byforming an ensemble, since prediction may be performed by synthesizingresults predicted from various forms of sub-neural network models,accuracy of result prediction may be further improved.

FIG. 14 is a block diagram for describing a training process of a neuralnetwork model according to an embodiment of the present invention.Referring to FIG. 14, the training process of a neural network modelaccording to an embodiment of the present invention may includeobtaining a data set (S1011), training a first model (that is, firstneural network model) and a second model (that is, second neural networkmodel) using the obtained data (S1031, S1033), validating the trainedfirst neural network model and second neural network model (S1051), anddetermining a final neural network model and obtaining parameters orvariables thereof (S1072).

Hereinafter, some embodiments of the training process of a neuralnetwork model will be described with reference to FIG. 14.

According to an embodiment of the present invention, a plurality ofsub-neural network models may obtain the same training data set andindividually generate output values. In this case, an ensemble of theplurality of sub-neural network models may be determined as a finalneural network model, and parameter values related to each of theplurality of sub-neural network models may be obtained as trainingresults. An output value of the final neural network model may be set toan average value of the output values by the sub-neural network models.Alternatively, in consideration of accuracy obtained as a result ofvalidating each of the sub-neural network models, the output value ofthe final neural network model may be set to a weighted average value ofthe output values of the sub-neural network models.

As a more specific example, when a neural network model includes a firstsub-neural network model and a second sub-neural network model,optimized parameter values of the first sub-neural network model andoptimized parameter values of the second sub-neural network model may beobtained by machine learning. In this case, an average value of outputvalues (for example, probability values related to specific diagnosisassistance information) obtained from the first sub-neural network modeland second sub-neural network model may be determined as an output valueof the final neural network model.

According to another embodiment of the present invention, accuracy ofindividual sub-neural network models may be evaluated on the basis ofoutput values by each of the plurality of sub-neural network models. Inthis case, any one of the plurality of sub-neural network models may beselected on the basis of the accuracy and determined as the final neuralnetwork model. A structure of the determined sub-neural network modeland parameter values of the determined sub-neural network model obtainedas a result of training may be stored.

As a more specific example, when a neural network model includes a firstsub-neural network model and a second sub-neural network model,accuracies of the first sub-neural network model and second sub-neuralnetwork model may be obtained, and a more accurate sub-neural networkmodel may be determined as the final neural network model.

According to still another embodiment of the present invention, one ormore sub-neural network models among a plurality of neural networkmodels may be combined, ensembles of the one or more combined sub-neuralnetwork models may be formed, and each ensemble may be evaluated,wherein a combination of sub-neural network models which forms the mostaccurate ensemble among the plurality of ensembles may be determined asa final neural network model. In this case, an ensemble may be formedfor all possible cases of selecting one or more of the plurality ofsub-neural network models, and a combination of sub-neural networkmodels which is evaluated to be the most accurate may be determined as afinal neural network model.

As a more specific example, when a neural network model includes a firstsub-neural network model and a second sub-neural network model, accuracyof the first sub-neural network model, accuracy of the second sub-neuralnetwork model, and accuracy of an ensemble of the first and secondsub-neural network models may be compared, and a sub-neural networkmodel combination of the most accurate case may be determined as a finalneural network model.

2.4 Embodiment 1—Control Method of Training Device

FIG. 15 is a view for describing a control method of a training deviceaccording to an embodiment of the present invention.

Referring to FIG. 15, the control method of a training device accordingto an embodiment of the present invention may include pre-processing afirst fundus image (S110), serializing the pre-processed first fundusimage (S130), and training a first neural network model (S150).

The control method of a training device according to an embodiment ofthe present invention may be a control method of a training deviceincluded in a system including a training device configured to obtain afirst training data set including a plurality of fundus images, processthe fundus images included in the first training data set, and train afirst neural network model using the first training data set and adiagnostic device configured to obtain a target fundus image forobtaining diagnosis assistance information and obtain the diagnosisassistance information on the basis of the target fundus image by usingthe trained first neural network model.

The pre-processing of the first fundus image (S110) may further includepre-processing the first fundus image so that the first fundus imageincluded in the first training data set is converted to a formatsuitable for training the first neural network model.

The control method of the training device according to an embodiment ofthe present invention may include the serializing of the pre-processedfirst fundus image (S130). The first fundus image may be serialized to aformat that facilitates training of the neural network model.

In this case, the training of the first neural network model (S150) mayfurther include training the first neural network model that classifiesthe target fundus image as a first label or a second label by using theserialized first fundus image.

The training device may obtain a second training data set which includesthe plurality of fundus images and at least partially differs from thefirst training data set and may train a second neural network modelusing the second training data set.

According to an embodiment of the present invention, the control methodof the training device may further include pre-processing a secondfundus image so that the second fundus image included in the second datatraining set is suitable for training the second neural network model,serializing the pre-processed second fundus image, and training thesecond neural network model that classifies the target fundus image as athird label or a fourth label by using the serialized second fundusimage.

FIG. 16 is a view for describing a control method of a training deviceaccording to an embodiment of the present invention. Referring to FIG.16, the control method of a training device according to an embodimentof the present invention may include pre-processing a second fundusimage (S210), serializing the pre-processed second fundus image (S230),and training a second neural network model (S250).

Although, for convenience of description, it has been depicted in FIG.16 that the pre-processing of the second fundus image, the serializingof the second fundus image, and the training using the second fundusimage may be performed subsequent to the pre-processing of the firstfundus image, the serializing of the first fundus image, and thetraining using the first fundus image, content of the invention is notlimited thereto.

The pre-processing of the second fundus image included in the secondtraining data set, the serializing of the second fundus image, and thetraining using the second fundus image may be performed independently ofthe above-described pre-processing of the first fundus image,serializing of the first fundus image, and training using the firstfundus image. The pre-processing of the second fundus image included inthe second training data set, the serializing of the second fundusimage, and the training using the second fundus image may be performedin parallel with the above-described pre-processing of the first fundusimage, serializing of the first fundus image, and training using thefirst fundus image. In other words, the pre-processing of the secondfundus image included in the second training data set, the serializingof the second fundus image, and the training using the second fundusimage are not necessarily performed subsequent or prior to theabove-described pre-processing of the first fundus image, serializing ofthe first fundus image, and training using the first fundus image. Theprocess related to the first fundus image and the process related to thesecond fundus image may be performed without dependence on each other.

First pre-processing performed in relation to the fundus image includedin the first training data set may be distinguished from secondpre-processing performed in relation to the fundus image included in thesecond training data set. For example, the first pre-processing may bepre-processing for emphasizing a blood vessel, and the secondpre-processing may be pre-processing for modulating color. Eachpre-processing may be determined in consideration of diagnosisassistance information desired to be obtained through each neuralnetwork model.

The control method of the training device according to an embodiment ofthe present invention may further include validating the first neuralnetwork model by evaluating accuracy of the trained first neural networkmodel by using a first validation data set that is at least partiallydistinguished from the first training data set and validating the secondneural network model by evaluating accuracy of the trained second neuralnetwork model by using a second validation data set that is at leastpartially distinguished from the second training data set. In this case,validation of the first neural network model and validation of thesecond neural network model may be performed independently of eachother.

Serialized first fundus images may be sequentially stored in a firstqueue, and a predetermined unit volume of the serialized fundus imagesstored in the first queue may be used each time in training the firstneural network model. Serialized second fundus images may besequentially stored in a second queue distinguished from the firstqueue, and a predetermined unit volume of the serialized fundus imagesstored in the second queue may be used each time in training the secondneural network model.

The first neural network model may include a first sub-neural networkmodel and a second sub-neural network model. In this case, classifying atarget fundus image as the first label or the second label may beperformed by simultaneously taking into consideration a first predictedvalue predicted by the first sub-neural network model and a secondpredicted value predicted by the second sub-neural network model.

The second neural network model may include a third sub-neural networkmodel and a fourth sub-neural network model. In this case, classifying atarget fundus image as the third label or the fourth label may beperformed by simultaneously taking into consideration a third predictedvalue predicted by the third sub-neural network model and a fourthpredicted value predicted by the fourth sub-neural network model.

The first training data set may include at least some of fundus imageslabeled with the first label, and the second training data set mayinclude at least some of fundus images labeled with the third label. Inthis case, the fundus images labeled with the first label may be thesame as at least some of the fundus images labeled with the third label.

The first label may be a normal label indicating that a patientcorresponding to the target fundus image is normal in relation to afirst finding, and the second label may be an abnormal label indicatingthat the patient is abnormal in relation to a second finding.

The pre-processing of the first fundus image may include cropping thefirst fundus image so that a reference aspect ratio is satisfied andchanging the size of the first fundus image.

The pre-processing of the first fundus image may further include, by aprocessing unit, applying a blood vessel emphasizing filter to thefundus image so that a blood vessel included in the first fundus imageis emphasized.

Serialized first fundus images may be sequentially stored in a queue,and a predetermined number of the serialized first fundus images storedin the queue may be used each time in training the first neural networkmodel. When the capacity of the serialized first fundus images whichhave not been used in the training of the first neural network model isreduced to a reference amount or lower, the queue may request forsupplementation of the serialized first fundus images.

The first finding may be any one of a finding of retinal hemorrhage, afinding of generation of retinal exudates, a finding of opacity ofcrystalline lens, and a finding of diabetic retinopathy.

FIG. 17 is a view for describing a control method of a training deviceaccording to an embodiment of the present invention.

Referring to FIG. 17, the control method of the training deviceaccording to an embodiment of the present invention may further includevalidating the first neural network model (S170) and updating the firstneural network model (S190).

The validating of the first neural network model (S170) may furtherinclude validating the first neural network model by evaluating accuracyof the trained first neural network model by using the first validationdata set that is at least partially distinguished from the firsttraining data set.

The updating of the first neural network model (S190) may furtherinclude updating the first neural network model by reflecting avalidation result obtained from the validating of the first neuralnetwork model (S170).

Meanwhile, the first neural network model may include a first sub-neuralnetwork model and a second sub-neural network model. In this case, thetraining of the first neural network model may include validating thefirst sub-neural network model using the first validation data set toobtain accuracy of the first sub-neural network model, validating thesecond sub-neural network model using the first validation data set toobtain accuracy of the second sub-neural network model, and comparingthe accuracy of the first sub-neural network model and the accuracy ofthe second sub-neural network model to determine a more accuratesub-neural network model as the final neural network model.

3. Diagnosis Assistance Process

According to an embodiment of the present invention, a diagnosisassistance process (or diagnostic process) in which diagnosis assistanceinformation is obtained using a neural network model may be provided. Asa specific example, by the diagnosis assistance process, diagnosisassistance information (for example, diagnostic information or findingsinformation) may be predicted through a diagnosis assistance neuralnetwork model trained using a fundus image.

The diagnosis assistance process which will be described below may beperformed by a diagnostic device.

3.1 Diagnostic Unit

According to an embodiment of the present invention, a diagnosticprocess may be performed by a diagnostic unit 200. The diagnostic unit200 may be provided in the above-described diagnostic device.

FIG. 18 is a view for describing a configuration of the diagnostic unit200 according to an embodiment of the present invention. Referring toFIG. 18, the diagnostic unit 200 may include a diagnosis requestobtaining module 210, a data processing module 230, a diagnostic module250, and an output module 270.

As will be described below, the modules may perform individual steps ofa data processing process and a training process. However, not all ofthe elements described with reference to FIG. 18 and functions performedby the elements are essential, and some elements may be added or omittedaccording to an aspect of diagnosis.

3.2 Obtaining Data and Diagnosis Request

The diagnostic device according to an embodiment of the presentinvention may obtain diagnosis target data and obtain diagnosisassistance information on the basis of the obtained diagnosis targetdata. The diagnosis target data may be image data. The obtaining of thedata and obtaining of a diagnosis request may be performed by thediagnosis request obtaining module of the above-described diagnosticunit.

FIG. 19 is a view for describing diagnosis target data TD according toan embodiment of the present invention. Referring to FIG. 19, thediagnosis target data TD may include a diagnosis target image TI anddiagnosis target patient information PI.

The diagnosis target image TI may be an image for obtaining diagnosisassistance information on a diagnosis target patient. For example, thediagnosis target image may be a fundus image. The diagnosis target imageTI may have any one format among JPG, PNG, DCM (DICOM), BMP, GIF, andTIFF.

The diagnosis patient information PI may be information for identifyinga patient to be diagnosed. Alternatively, the diagnosis patientinformation PI may be characteristic information of a patient or animage to be diagnosed. For example, the diagnosis patient information PImay include information such as the date and time of imaging and imagingequipment of an image to be diagnosed or information such as anidentification (ID) number, an ID, name, age, or weight of a patient tobe diagnosed. When the image to be diagnosed is a fundus image, thediagnosis patient information PI may further include eye-relatedinformation such as left eye/right eye information on whether thecorresponding fundus image is an image of the left eye or an image ofthe right eye.

The diagnostic device may obtain a diagnosis request. The diagnosticdevice may obtain diagnosis target data together with the diagnosisrequest. When the diagnosis request is obtained, the diagnostic devicemay obtain diagnosis assistance information using a trained diagnosisassistance neural network model. The diagnostic device may obtain adiagnosis request from a client device. Alternatively, the diagnosticdevice may obtain a diagnosis request from a user through aseparately-provided input means.

3.3 Date Processing Process

Obtained data may be processed. Data processing may be performed by thedata processing module of the above-described diagnostic unit.

Generally, a data processing process may be performed similar to thedata processing process in the above-described training process.Hereinafter, the data processing process in the diagnostic process willbe described focusing on differences from the data processing process inthe training process.

In the diagnostic process, the diagnostic device may obtain data as inthe training process. In this case, the obtained data may have the sameformat as the data obtained in the training process. For example, whenthe training device has trained a diagnosis assistance neural networkmodel using image data in the DCM format in the training process, thediagnostic device may obtain the DCM image and obtain diagnosisassistance information using the trained neural network model.

In the diagnostic process, the obtained image to be diagnosed may beresized similar to the image data used in the training process. Toefficiently perform prediction of diagnosis assistance informationthrough the trained diagnosis assistance neural network model, the formof the image to be diagnosed may be adjusted to have a suitable volume,size, and/or aspect ratio.

For example, when an image to be diagnosed is a fundus image, resizingof the image such as removing an unnecessary portion of the image orreducing the size of the image may be performed to predict diagnosticinformation on the basis of the fundus image.

In the diagnostic process, similar to the image data used in thetraining process, a pre-processing filter may be applied to the obtainedimage to be diagnosed. A suitable filter may be applied to the image tobe diagnosed so that accuracy of prediction of diagnosis assistanceinformation through a trained diagnosis assistance neural network modelis further improved.

For example, when an image to be diagnosed is a fundus image,pre-processing that facilitates prediction of correct diagnosticinformation, for example, image pre-processing that causes a bloodvessel to be emphasized or image pre-processing that causes a specificcolor to be emphasized or weakened, may be applied to the image to bediagnosed.

In the diagnostic process, similar to the image data used in thetraining process, the obtained image to be diagnosed may be serialized.The image to be diagnosed may be converted to a form that facilitatesdriving of a diagnostic model in a specific work frame or may beserialized.

The serializing of the image to be diagnosed may be omitted. This may bebecause, in the diagnostic process, the number of data processed at onetime by a processor is not large unlike in the training process, andthus the burden on data processing speed is relatively small.

In the diagnostic process, similar to the image data used in thetraining process, the obtained image to be diagnosed may be stored in aqueue. However, since the number of data being processed is smaller inthe diagnostic process in comparison to that in the training process,storing data in a queue may also be omitted.

Meanwhile, since an increase in the number of data is not required inthe diagnostic process, it is preferable that, in order to obtainaccurate diagnosis assistance information, the process of dataaugmentation or image augmentation is not used, unlike in the trainingprocess.

3.4 Diagnostic Process

According to an embodiment of the present invention, a diagnosticprocess using a trained neural network model may be disclosed. Thediagnostic process may be performed by the above-described diagnosticdevice. The diagnostic process may be performed by the above-describeddiagnostic server. The diagnostic process may be performed by thecontrol unit of the above-described diagnostic device. The diagnosticprocess may be performed by the diagnostic module of the above-describeddiagnostic unit.

FIG. 20 is a view for describing a diagnostic process according to anembodiment of the present invention. Referring to FIG. 20, thediagnostic process may include obtaining diagnosis target data (S2010),using a trained neural network model (S2030), and obtaining andoutputting a result corresponding to the obtained diagnosis target data(S2050). However, data processing may be selectively performed.

Hereinafter, each step of the diagnostic process will be described withreference to FIG. 20.

3.4.1 Data Input

According to an embodiment of the present invention, the diagnosticmodule may obtain diagnosis target data. The obtained data may be dataprocessed as described above. For example, the obtained data may be apatient's fundus image data to which pre-processing that causes the sizeto be adjusted and a blood vessel to be emphasized is applied. Accordingto an embodiment of the present invention, a left eye image and a righteye image of a single patient may be input together as diagnosis targetdata.

3.4.2 Data Classification

A diagnosis assistance neural network model provided in the form of aclassifier may classify input diagnosis target images into a positiveclass or a negative class in relation to a predetermined label.

A trained diagnosis assistance neural network model may receivediagnosis target data and output a predicted label. The traineddiagnosis assistance neural network model may output a predicted valueof diagnosis assistance information. Diagnosis assistance informationmay be obtained using the trained diagnosis assistance neural networkmodel. The diagnosis assistance information may be determined on thebasis of the predicted label.

For example, the diagnosis assistance neural network model may predictdiagnostic information (that is, information on the presence of adisease) or findings information (that is, information on the presenceof abnormal findings) related to an eye disease or a systemic disease ofthe patient. In this case, the diagnostic information or findingsinformation may be output in the form of a probability. For example, theprobability that the patient has a specific disease or the probabilitythat there may be a specific abnormal finding in the patient's fundusimage may be output. When a diagnosis assistance neural network modelprovided in the form of a classifier is used, a predicted label may bedetermined in consideration of whether an output probability value (orpredicted score) exceeds a threshold value.

As a specific example, a diagnosis assistance neural network model mayoutput a probability value with respect to the presence of diabeticretinopathy in a patient with the patient's fundus image as a diagnosistarget image. When a diagnosis assistance neural network model in theform of a classifier that assumes 1 as normal is used, a patient'sfundus image may be input to the diagnosis assistance neural networkmodel, and in relation to whether the patient has diabetic retinopathy,a normal:abnormal probability value may be obtained in the form of0.74:0.26 or the like.

Although the case in which data is classified using the diagnosisassistance neural network model in the form of a classifier has beendescribed herein, the present invention is not limited thereto, and aspecific diagnosis assistance numerical value (for example, bloodpressure or the like) may also be predicted using a diagnosis assistanceneural network model implemented in the form of a regression model.

According to another embodiment of the present invention, suitabilityinformation on an image may be obtained. The suitability information mayindicate whether a diagnosis target image is suitable for obtainingdiagnosis assistance information using a diagnosis assistance neuralnetwork model.

The suitability information of an image may be quality information. Thequality information or suitability information may indicate whether adiagnosis target image reaches a reference level.

For example, when a diagnosis target image has a defect due to a defectof imaging equipment or an influence of an illumination during imaging,a result indicating that the diagnosis target image is unsuitable may beoutput as suitability information of the corresponding diagnosis targetimage. When a diagnosis target image includes noise at a predeterminedlevel or higher, the diagnosis target image may be determined as beingunsuitable.

The suitability information may be a value predicted using a neuralnetwork model. Alternatively, the suitability information may beinformation obtained through a separate image analysis process.

According to an embodiment, even when an image is classified asunsuitable, diagnosis assistance information may be obtained on thebasis of the unsuitable image.

According to an embodiment, an image classified as unsuitable may bereexamined by a diagnosis assistance neural network model.

In this case, the diagnosis assistance neural network model thatperforms the reexamination may differ from a diagnosis assistance neuralnetwork model that performs initial examination. For example, thediagnostic device may store a first diagnosis assistance neural networkmodel and a second diagnosis assistance neural network model, and animage classified as unsuitable through the first diagnosis assistanceneural network model may be examined through the second diagnosisassistance neural network model.

According to still another embodiment of the present invention, a classactivation map (CAM) may be obtained from a trained neural networkmodel. Diagnosis assistance information may include a CAM. The CAM maybe obtained together with other diagnosis assistance information.

The CAM may be obtained optionally. For example, the CAM may beextracted and/or output when diagnostic information or findingsinformation obtained by a diagnosis assistance model is classified intoan abnormal class.

3.5 Output of Diagnosis Assistance Information

Diagnosis assistance information may be determined on the basis of alabel predicted from a diagnosis assistance neural network model.

Output of diagnosis assistance information may be performed by theoutput module of the above-described diagnostic unit. Diagnosisassistance information may be output from the diagnostic device to aclient device. Diagnosis assistance information may be output from thediagnostic device to a server device. Diagnosis assistance informationmay be stored in the diagnostic device or diagnostic server. Diagnosisassistance information may be stored in a separately-provided serverdevice or the like.

Diagnosis assistance information may be managed by being formed into adatabase. For example, obtained diagnosis assistance information may bestored and managed together with a diagnosis target image of a subjectaccording to an identification number of the corresponding subject. Inthis case, the diagnosis target image and diagnosis assistanceinformation of the patient may be managed in chronological order. Bymanaging the diagnosis assistance information and diagnosis target imagein time series, tracking personal diagnostic information and managinghistory thereof may be facilitated.

Diagnosis assistance information may be provided to a user. Thediagnosis assistance information may be provided to the user through anoutput means of a diagnostic device or client device. The diagnosisassistance information may be output through a visual or aural outputmeans provided in the diagnostic device or client device so that theuser may recognize the diagnosis assistance information.

According to an embodiment of the present invention, an interface foreffectively providing diagnosis assistance information to a user may beprovided. Such a user interface will be described in more detail belowin Section “5. User interface.”

When a CAM is obtained by a neural network model, an image of the CAMmay be provided together. The image of the CAM may be selectivelyprovided. For example, the CAM image may not be provided when diagnosticinformation obtained through a diagnosis assistance neural network modelis normal findings information or normal diagnostic information, and theCAM image may be provided together for more accurate clinical diagnosiswhen the obtained diagnostic information is abnormal findingsinformation or abnormal diagnostic information.

When an image is classified as unsuitable, suitability information ofthe image may be provided together. For example, when an image isclassified as unsuitable, diagnosis assistance information and“unsuitable” judgment information obtained according to thecorresponding image may be provided together.

A diagnosis target image that has been judged to be unsuitable may beclassified as an image to be retaken. In this case, a retake guide for atarget patient of the image classified as an image to be retaken may beprovided together with the suitability information.

Meanwhile, in response to providing of diagnosis assistance informationobtained through a neural network model, feedback related to training ofthe neural network model may be obtained. For example, feedback foradjusting a parameter or hyperparameter related to training of theneural network model may be obtained. The feedback may be obtainedthrough a user input unit provided in the diagnostic device or clientdevice.

According to an embodiment of the present invention, diagnosisassistance information corresponding to a diagnosis target image mayinclude level information. The level information may be selected among aplurality of levels. The level information may be determined on thebasis of diagnostic information and/or findings information obtainedthrough a neural network model. The level information may be determinedin consideration of suitability information or quality information of adiagnosis target image. When a neural network model is a classifiermodel that performs multiclass classification, the level information maybe determined in consideration of a class into which a diagnosis targetimage is classified by the neural network model. When a neural networkmodel is a regression model that outputs a numerical value related to aspecific disease, the level information may be determined inconsideration of the output numerical value.

For example, diagnosis assistance information obtained corresponding toa diagnosis target image may include any one level information selectedfrom a first level information and a second level information. Whenabnormal findings information or abnormal diagnostic information isobtained through a neural network model, the first level information maybe selected as the level information. When abnormal findings informationor abnormal diagnostic information is not obtained through a neuralnetwork model, the second level information may be selected as the levelinformation. Alternatively, the first level information may be selectedas the level information when a numerical value obtained through aneural network model exceeds a reference numerical value, and the secondlevel information may be selected as the level information when theobtained numerical value is less than the reference numerical value. Thefirst level information may indicate that strong abnormal information ispresent in a diagnosis target image compared with the second levelinformation.

Meanwhile, a third level information may be selected as the levelinformation when the quality of a diagnosis target image is determinedto a reference quality or lower using image analysis or a neural networkmodel. Alternatively, diagnosis assistance information may include thethird level information together with the first or second levelinformation.

When diagnosis assistance information includes the first levelinformation, a first user guide may be output through an output means.The first user guide may indicate that a more precise test is requiredfor a testee(patient) corresponding to the diagnosis assistanceinformation. For example, the first user guide may indicate thatsecondary diagnosis (for example, diagnosis in a separate medicalinstitution or a hospital transfer procedure) is required for thepatient. Alternatively, the first user guide may indicate treatmentrequired for the patient. As a specific example, when abnormalinformation on macular degeneration of the patient is obtained bydiagnosis assistance information, the first user guide may includeinjection prescription and a guide on a hospital transfer procedure (forexample, a list of hospitals to which transfer is possible) related tothe patient.

When diagnosis assistance information includes the second levelinformation, a second user guide may be output through an output means.The second user guide may include future care plans related to thepatient corresponding to the diagnosis assistance information. Forexample, the second user guide may indicate the time of next visit andthe next medical course.

When diagnosis target information includes the third level information,a third user guide may be output through an output means. The third userguide may indicate that a diagnosis target image has to be retaken. Thethird user guide may include information on the quality of the diagnosistarget image. For example, the third user guide may include informationon an artifact present in a diagnosis target image (for example, whetherthe artifact is a bright artifact or a dark artifact, or the degreethereof).

4. Diagnosis Assistance System for Multiple Labels

According to an embodiment of the present invention, a diagnosisassistance system for performing prediction on a plurality of labels(for example, a plurality of diagnosis assistance information) may beprovided. For this, a diagnosis assistance neural network of theabove-mentioned diagnosis assistance system may be designed to performprediction on a plurality of labels.

Alternatively, in the above-mentioned diagnosis assistance system, aplurality of diagnosis assistance neural networks that performprediction on different labels may be used in parallel. Hereinafter,such a parallel diagnosis assistance system will be described.

4.1 Configuration of Parallel Diagnosis Assistance System

According to an embodiment of the present invention, a paralleldiagnosis assistance system for obtaining a plurality of diagnosisassistance information may be provided. The parallel diagnosisassistance system may train a plurality of neural network models forobtaining a plurality of diagnosis assistance information and obtain theplurality of diagnosis assistance information using the trainedplurality of neural network models.

For example, the parallel diagnosis assistance system may train, on thebasis of fundus images, a first neural network model that obtains afirst diagnosis assistance information related to the presence of an eyedisease of a patient and a second neural network model that obtains asecond diagnosis assistance information related to the presence of asystemic disease of the patient and may output the diagnosis assistanceinformation related to the presence of an eye disease and the presenceof a systemic disease of the patient by using the trained first neuralnetwork model and the second neural network model.

FIGS. 21 and 22 are views for describing a parallel diagnosis assistancesystem according to some embodiments of the present invention. Referringto FIGS. 21 and 22, the parallel diagnosis assistance system may includea plurality of training units.

Referring to FIG. 21, a parallel diagnosis assistance system 30according to an embodiment of the present invention may include atraining device 1000, a diagnostic device 2000, and a client device3000. In this case, the training device 1000 may include a plurality oftraining units. For example, the training device 1000 may include afirst training unit 100 a and a second training unit 100 b.

Referring to FIG. 22, a parallel diagnosis assistance system 40according to an embodiment of the present invention may include a firsttraining device 1000 a, a second training device 1000 b, a diagnosticdevice 2000, and a client device 3000. The first training device 1000 amay include a first training unit 100 a. The second training device 1000b may include a second training unit 100 b.

Referring to FIGS. 21 and 22, the first training unit 100 a may obtain afirst data set and output a first parameter set of a first neuralnetwork model obtained as a result of training the first neural networkmodel. The second training unit 100 b may obtain a second data set andoutput a second parameter set of a second neural network model obtainedas a result of training the second neural network model.

The diagnostic device 2000 may include a diagnostic unit 200.Description similar to that given above with reference to FIG. 1 may beapplied to the diagnostic device 2000 and the diagnostic unit 200. Thediagnostic unit 200 may obtain a first diagnosis assistance informationand a second diagnosis assistance information using the trained firstneural network model and second neural network model through the firsttraining unit 100 a and the second training unit 100 b. The diagnosticunit 200 may store parameters of the trained first neural network modeland parameters of the trained second neural network model obtained fromthe first training unit 100 a and the second training unit 100 b.

The client device 3000 may include a data obtaining unit, e.g., animaging unit 300. However, the imaging unit 300 may be substituted withother data obtaining means used for obtaining diagnosis assistanceinformation. The client device may transmit a diagnosis request anddiagnosis target data (for example, a fundus image obtained by theimaging unit) to the diagnostic device. In response to the transmittingof the diagnosis request, the client device 3000 may obtain, from thediagnostic device, a plurality of diagnosis assistance informationaccording to the transmitted diagnosis target data.

Meanwhile, although the case in which the diagnosis assistance system 40includes the first training unit 100 a and the second training unit 100b has been described above with reference to FIGS. 21 and 22, content ofthe invention is not limited thereto. According to another embodiment ofthe present invention, a training device may include a training unitconfigured to obtain three or more different diagnosis assistanceinformation. Alternatively, a diagnosis assistance system may alsoinclude a plurality of training devices configured to obtain differentdiagnosis assistance information

The operations of the training device, the diagnostic device, and theclient device will be described in more detail below.

4.2 Parallel Training Process

According to an embodiment of the present invention, a plurality ofneural network models may be trained. Training processes for trainingthe respective neural network models may be performed in parallel.

4.2.1 Parallel Training Units

Training processes may be performed by a plurality of training units.The training processes may be performed independently of each other. Theplurality of training units may be provided in a single training deviceor respectively provided in a plurality of training devices.

FIG. 23 is a view for describing a configuration of a training deviceincluding a plurality of training units according to an embodiment ofthe present invention. The configuration and operation of each of thefirst training unit 100 a and the second training unit 100 b may beimplemented similar to those described above with reference to FIG. 9.

Referring to FIG. 23, a process of a neural network model according toan embodiment of the present invention may be performed by a trainingdevice 1000 including a first training unit 100 a which includes a firstdata processing module 110 a, a first queue module 130 a, a firsttraining module 150 a, and a first training result obtaining module 170a and a second training unit 100 b which includes a second dataprocessing module 110 b, a second queue module 130 b, a second trainingmodule 150 b, and a second training result obtaining module 170 b.

Referring to FIG. 23, a training process of a neural network modelaccording to an embodiment of the present invention may be performed byeach of the first training unit 100 a and the second training unit 100b. The first training unit 100 a and the second training unit 100 b mayindependently perform training of the first neural network model and thesecond neural network model. Referring to FIG. 23, the first trainingunit 100 a and the second training unit 100 b may be provided in theabove-described training device. Alternatively, the first training unitand the second training unit may also be provided in different trainingdevices.

4.2.2 Obtaining Parallel Data

According to an embodiment of the present invention, a plurality oftraining units may obtain data. The plurality of training units mayobtain different data sets. Alternatively, the plurality of trainingunits may also obtain the same data set. According to circumstances, theplurality of training units may also obtain partially common data sets.The data sets may be fundus image data sets.

A first training unit may obtain a first data set, and a second trainingunit may obtain a second data set. The first data set and the seconddata set may be distinguished from each other. The first data set andthe second data set may be partially common. The first data set and thesecond data set may be labeled fundus image data sets.

The first data set may include data labeled as normal in relation to afirst feature and data labeled as abnormal in relation to the firstfeature. For example, the first data set may include a fundus imagelabeled as normal and a fundus image labeled as abnormal in relation tothe opacity of crystalline lens.

The second data set may include data labeled as normal in relation to asecond feature (distinguished from the first feature) and data labeledas abnormal in relation to the second feature. For example, the seconddata set may include a fundus image labeled as normal and a fundus imagelabeled as abnormal in relation to diabetic retinopathy.

The data labeled as normal in relation to the first feature and datalabeled as normal in relation to the second feature respectivelyincluded in the first data set and the second data set may be common.For example, the first data set may include a fundus image labeled asnormal and a fundus image labeled as abnormal in relation to the opacityof crystalline lens, and the second data set may include a fundus imagelabeled as normal and a fundus image labeled as abnormal in relation todiabetic retinopathy, wherein the fundus image labeled as normal inrelation to the opacity of crystalline lens included in the first dataset and the fundus image labeled as normal in relation to diabeticretinopathy included in the second data set may be common.

Alternatively, the data labeled as abnormal in relation to the firstfeature and the data labeled as abnormal in relation to the secondfeature respectively included in the first data set and the second dataset may also be common. That is, data labeled in relation to a pluralityof features may be used in training a neural network model in relationto the plurality of features.

Meanwhile, the first data set may be a fundus image data set capturedusing a first method, and the second data set may be a fundus image dataset captured using a second method. The first method and the secondmethod may be any one method selected from red-free imaging, panoramicimaging, autofluorescence imaging, infrared imaging, and the like.

A data set used in each training unit may be determined in considerationof diagnosis assistance information obtained by a trained neural networkmodel. For example, when the first training unit trains a first neuralnetwork model which desires to obtain diagnosis assistance informationrelated to abnormal findings of the retina (for example, microaneurysms,exudates, and the like), the first training unit may obtain a firstfundus image data set captured by red-free imaging. Alternatively, whenthe second training unit trains a second neural network model whichdesires to obtain diagnosis assistance information related to maculardegeneration, the second training unit may obtain a second fundus imagedata set captured by autofluorescence imaging.

4.2.3 Parallel Data Processing

The plurality of training units may process obtained data. As describedabove in Section “2.2 Data processing process,” each training unit mayprocess data by applying one or more of image resizing, a pre-processingfilter, image augmentation, and image serialization to obtained data.The first data processing module of the first training unit may processa first data set, and the second data processing module of the secondtraining unit may process a second data set.

The first training unit and second training unit included in theplurality of training units may differently process obtained data setsin consideration of diagnosis assistance information obtained fromneural network models respectively trained by the first training unitand the second training unit. For example, to train a first neuralnetwork model for obtaining a first diagnosis assistance informationrelated to hypertension, the first training unit may performpre-processing that causes blood vessels to be emphasized in fundusimages included in the first fundus image data set. Alternatively, totrain a second neural network model for obtaining a second diagnosisassistance information related to abnormal findings on exudates,microaneurysms, and the like of the retina, the second training unit mayperform pre-processing that causes fundus images included in the secondfundus image data set to be converted to red-free images.

4.2.4 Parallel Queue

The plurality of training units may store data in a queue. As describedabove in Section “2.2.6 Queue,” each training unit may store processeddata in a queue and transmit the processed data to the training module.For example, the first training unit may store a first data set in afirst queue module and provide the first data set to a first trainingmodule sequentially or randomly. The second training module may store asecond data set in a second queue module and provide the second data setto a second training module sequentially or randomly.

4.2.5 Parallel Training Process

The plurality of training units may train a neural network model. Thetraining modules may independently train diagnosis assistance neuralnetwork models that perform prediction on different labels usingtraining data sets. A first training module of the first training unitmay train the first neural network model, and a second training moduleof the second training unit may train the second neural network model.

The plurality of diagnosis assistance neural network models may betrained in parallel and/or independently. By training models to performprediction on different labels through the plurality of neural networkmodels in this way, accuracy of prediction on each label may beimproved, and efficiency of the prediction operation may be enhanced.

Each diagnosis assistance neural network model may be provided similarto that described above in Section “2.3.2 Model design.” Eachsub-training process may be performed similar to that described above inSections 2.3.1 to 2.3.5.

A parallel training process according to an embodiment of the presentinvention may include training diagnosis assistance neural networkmodels that predict different labels. The first training unit may traina first diagnosis assistance neural network model that predicts a firstlabel. The second training unit may train a second diagnosis assistanceneural network model that predicts a second label.

The first training unit may obtain a first data set and train the firstdiagnosis assistance neural network model that predicts the first label.For example, the first training unit may train the first diagnosisassistance neural network model that predicts the presence of maculardegeneration of a patient from a fundus image by using a fundus imagetraining data set labeled in relation to the presence of maculardegeneration.

The second training unit may obtain a second data set and train thesecond diagnosis assistance neural network model that predicts thesecond label. For example, the second training unit may train the seconddiagnosis assistance neural network model that predicts the presence ofdiabetic retinopathy of a patient from a fundus image by using a fundusimage training data set labeled in relation to the presence of diabeticretinopathy.

The training process of a neural network model will be described in moredetail below with reference to FIGS. 24 and 25.

FIG. 24 is a view for describing a parallel training process accordingto an embodiment of the present invention. The parallel training processmay be applied to all of the cases in which the parallel diagnosisassistance system is implemented as shown in FIG. 21, implemented asshown in FIG. 22, and implemented in other forms. However, forconvenience of description, description will be given below on the basisof the parallel diagnosis assistance system implemented as shown in FIG.21.

Referring to FIG. 24, the parallel training process may include aplurality of sub-training processes that respectively train a pluralityof diagnosis assistance neural network models that predict differentlabels. The parallel training process may include a first sub-trainingprocess that trains a first neural network model and a secondsub-training process that trains a second neural network model.

For example, the first sub-training process may be performed byobtaining a first data (S1010 a), using a first neural network model(S1030 a), validating the first model (that is, first diagnosisassistance neural network model) (S1050 a), and obtaining parameters ofthe first neural network model (S1070 a). The second sub-trainingprocess may be performed by obtaining a second data (S1010 b), using asecond neural network model (1030 b), validating the second neuralnetwork model (that is, second diagnosis assistance neural networkmodel) (S1050 b), and obtaining parameters of the second neural networkmodel (S1070 b).

A sub-training process may include training a neural network model byinputting training data into a sub-neural network model, comparing alabel value obtained by output with the input training data to validatethe model, and reflecting a validation result back to the sub-neuralnetwork model.

Each sub-training process may include obtaining result values using aneural network model to which arbitrary weight values are assigned,comparing the obtained result values with label values of training data,and performing backpropagation according to errors therebetween tooptimize the weight values.

In each sub-training process, a diagnosis assistance neural networkmodel may be validated through a validation data set distinguished froma training data set. Validation data sets for validating a first neuralnetwork model and a second neural network model may be distinguished.

The plurality of training units may obtain training results. Eachtraining result obtaining module may obtain information on neuralnetwork models trained from the training modules. Each training resultobtaining module may obtain parameter values of neural network modelstrained from the training units. A first training result obtainingmodule of the first training unit may obtain a first parameter set of afirst neural network model trained from a first training module. Asecond training result obtaining module of the second training unit mayobtain a second parameter set of a second neural network model trainedfrom a second training module.

By each sub-training process, optimized parameter values, that is, aparameter set, of a trained neural network model may be obtained. Astraining is performed using more training data sets, more suitableparameter values may be obtained.

A first parameter set of a first diagnosis assistance neural networkmodel trained by a first sub-training process may be obtained. A secondparameter set of a second diagnosis assistance neural network modeltrained by a second sub-training process may be obtained. As training issufficiently performed, optimized values of weights and/or bias of thefirst diagnosis assistance neural network and the second diagnosisassistance neural network may be obtained.

The obtained parameter set of each neural network model may be stored inthe training device and/or the diagnostic device (or server). The firstparameter set of the first diagnosis assistance neural network and thesecond parameter set of the second diagnosis assistance neural networkmay be stored together or separately. A parameter set of each trainedneural network model may also be updated by feedback obtained from thediagnostic device or client device.

4.2.6 Parallel Ensemble Training Process

Even when a plurality of neural network models are trained in parallel,The above-described ensemble form of model training may be used. Eachsub-training process may include training a plurality of sub-neuralnetwork models. The plurality of sub-models may have different layerstructures. Hereinafter, unless particularly mentioned otherwise,description similar to that given above in Section 2.3.7 may be applied.

When a plurality of diagnosis assistance neural network models aretrained in parallel, some sub-training processes among the sub-trainingprocesses that train the diagnosis assistance neural network models maytrain a single model, and other sub-training processes may train aplurality of sub-models together.

Since models are trained using ensembles in each sub-training process,more optimized forms of neural network models may be obtained in eachsub-training process, and error in prediction may be reduced.

FIG. 25 is a view for describing the parallel training process accordingto another embodiment of the present invention. Referring to FIG. 25,each training process may include training a plurality of sub-neuralnetwork models.

Referring to FIG. 25, a first sub-training process may be performed byobtaining a first data S1011 a, using a first-first(1-1) neural networkmodel and a first-second(1-2) neural network model (S1031 a, S1033 a),validating the first-first(1-1) neural network model and thefirst-second(1-2) neural network model (S1051 a), and determining afinal form of the first neural network model and parameters thereof(S1071 a). A second sub-training process may be performed by obtaining asecond data (S1011 b), using a second-first(2-1) neural network modeland a second-second(2-2) neural network model (S1031 b, S1033 b),validating the second-first(2-1) neural network model and thesecond-second(2-2) neural network model (S1051 b), and determining afinal form of the second model (that is, the second diagnosis assistanceneural network model) and parameters thereof (S1071 b).

The first neural network trained in the first sub-training process mayinclude the first-first(1-1) neural network model and thefirst-second(1-2) neural network model. The first-first(1-1) neuralnetwork model and the first-second(1-2) neural network model may beprovided in different layer structures. Each of the first-first(1-1)neural network model and the first-second(1-2) neural network model mayobtain a first data set and output predicted labels. Alternatively, alabel predicted by an ensemble of the first-first(1-1) neural networkmodel and the first-second(1-2) neural network model may be determinedas a final predicted label.

In this case, the first-first(1-1) neural network model and thefirst-second(1-2) neural network model may be validated using avalidation data set, and a more accurate neural network model may bedetermined as a final neural network model. Alternatively, thefirst-first(1-1) neural network model, the first-second(1-2) neuralnetwork model, and the ensemble of the first-first(1-1) neural networkmodel and the first-second(1-2) neural network model may be validated,and a neural network model form of the most accurate case may bedetermined as a final first neural network model.

For the second sub-training process, likewise, the most accurate form ofneural network among the second-first(2-1) neural network model, thesecond-second(2-2) neural network model, and the ensemble of thesecond-first(2-1) neural network model and the second-second(2-2) neuralnetwork model may be determined as the final second model (that is,second diagnosis assistance neural network model).

Meanwhile, although, for convenience of description, the case in whicheach sub-training process includes two sub-models has been describedabove with reference to FIG. 25, this is merely an example, and thepresent invention is not limited thereto. A neural network model trainedin each sub-training process may only include a single neural networkmodel or include three or more sub-models.

4.3 Parallel Diagnostic Process

According to an embodiment of the present invention, a diagnosticprocess for obtaining a plurality of diagnosis assistance informationmay be provided. The diagnostic process for obtaining the plurality ofdiagnosis assistance information may be implemented in the form of aparallel diagnosis assistance process including a plurality ofdiagnostic processes which are independent from each other.

4.3.1 Parallel Diagnostic Unit

According to an embodiment of the present invention, a diagnosisassistance process may be performed by a plurality of diagnosticmodules. Each diagnosis assistance process may be independentlyperformed.

FIG. 26 is a block diagram for describing a diagnostic unit 200according to an embodiment of the present invention.

Referring to FIG. 26, the diagnostic unit 200 according to an embodimentof the present invention may include a diagnosis request obtainingmodule 211, a data processing module 231, a first diagnostic module 251,a second diagnostic module 253, and an output module 271. Unlessparticularly mentioned otherwise, each module of the diagnostic unit 200may operate similar to the diagnostic module of the diagnostic unitillustrated in FIG. 18.

In FIG. 26, the diagnosis request obtaining module 211, the dataprocessing module 231, and the output module 271 have been illustratedas being common even when the diagnostic unit 200 includes a pluralityof diagnostic modules, but the present invention is not limited to sucha configuration. The diagnosis request obtaining module, the dataprocessing module, and/or the output module may also be provided inplural. The plurality of diagnosis request obtaining modules, dataprocessing modules, and/or output modules may also operate in parallel.

For example, the diagnostic unit 200 may include a first data processingmodule configured to perform first processing of an input diagnosistarget image and a second data processing module configured to performsecond processing of the diagnosis target image, the first diagnosticmodule may obtain a first diagnosis assistance information on the basisof the diagnosis target image on which the first processing has beenperformed, and the second diagnostic module may obtain a seconddiagnosis assistance information on the basis of the diagnosis targetimage on which the second processing has been performed. The firstprocessing and/or second processing may be any one selected from imageresizing, image color modulation, blur filter application, blood vesselemphasizing process, red-free conversion, partial region cropping, andextraction of some elements.

The plurality of diagnostic modules may obtain different diagnosisassistance information. The plurality of diagnostic modules may obtaindiagnosis assistance information using different diagnosis assistanceneural network models. For example, the first diagnostic module mayobtain a first diagnosis assistance information related to the presenceof an eye disease of a patient by using a first neural network modelthat predicts the presence of an eye disease of the patient, and thesecond diagnostic module may obtain a second diagnosis assistanceinformation related to the presence of a systemic disease of a patientby using a second neural network model that predicts the presence of asystemic disease of the patient.

As a more specific example, the first diagnostic module may obtain afirst diagnosis assistance information related to the presence ofdiabetic retinopathy of the patient using a first diagnosis assistanceneural network model that predicts the presence of diabetic retinopathyof the patient, and the second diagnostic module may obtain a seconddiagnosis assistance information related to the presence of hypertensionusing a second diagnosis assistance neural network model that predictsthe presence of hypertension of the patient.

4.3.2 Parallel Diagnostic Process

A diagnosis assistance process according to an embodiment of the presentinvention may include a plurality of sub-diagnostic processes. Eachsub-diagnostic process may be performed using different diagnosisassistance neural network models. Each sub-diagnostic process may beperformed in different diagnostic modules. For example, a firstdiagnostic module may perform a first sub-diagnostic process thatobtains a first diagnosis assistance information through a firstdiagnosis assistance neural network model. Alternatively, a seconddiagnostic module may perform a second sub-diagnostic process thatobtains a second diagnosis assistance information through a seconddiagnosis assistance neural network model.

The plurality of trained neural network models may output a predictedlabel or probability with diagnosis target data as input. Each neuralnetwork model may be provided in the form of a classifier and mayclassify input diagnosis target data as a predetermined label. In thiscase, the plurality of neural network models may be provided in forms ofclassifiers that are trained in relation to different characteristics.Each neural network model may classify diagnosis target data asdescribed above in Section 3.4.2.

Meanwhile, a CAM may be obtained from each diagnosis assistance neuralnetwork model. The CAM may be obtained selectively. The CAM may beextracted when a predetermined condition is satisfied. For example, whena first diagnosis assistance information indicates that the patient isabnormal in relation to a first characteristic, a first CAM may beobtained from a first diagnosis assistance neural network model.

FIG. 27 is a view for describing a diagnosis assistance processaccording to an embodiment of the present invention.

Referring to FIG. 27, a diagnosis assistance process according to anembodiment of the present invention may include obtaining diagnosistarget data (S2011), using a first diagnosis assistance neural networkmodel and a second diagnosis assistance neural network model (S2031 a,S2031 b), and obtaining diagnosis assistance information according todiagnosis target data (S2051). The diagnosis target data may beprocessed data.

The diagnosis assistance process according to an embodiment of thepresent invention may include obtaining a first diagnosis assistanceinformation through the trained first diagnosis assistance neuralnetwork model and obtaining a second diagnosis assistance informationthrough the trained second diagnosis assistance neural network model.The first diagnosis assistance neural network model and the seconddiagnosis assistance neural network model may obtain the first diagnosisassistance information and the second diagnosis assistance informationrespectively, on the basis of the same diagnosis target data.

For example, the first diagnosis assistance neural network model and thesecond diagnosis assistance neural network model may respectively obtaina first diagnosis assistance information related to the presence ofmacular degeneration of the patient and a second diagnosis assistanceinformation related to the presence of diabetic retinopathy of thepatient on the basis of a diagnosis target fundus image.

In addition, unless particularly described otherwise, the diagnosisassistance process described with reference to FIG. 27 may beimplemented similar to the diagnosis assistance process described abovewith reference to FIG. 20.

4.3.3 Output of Diagnosis Assistance Information

According to an embodiment of the present invention, diagnosisassistance information may be obtained by a parallel diagnosisassistance process. The obtained diagnosis assistance information may bestored in the diagnostic device, server device, and/or client device.The obtained diagnosis assistance information may be transmitted to anexternal device.

A plurality of diagnosis assistance information may respectivelyindicate a plurality of labels predicted by a plurality of diagnosisassistance neural network models. The plurality of diagnosis assistanceinformation may respectively correspond to the plurality of labelspredicted by the plurality of diagnosis assistance neural networkmodels. Alternatively, diagnosis assistance information may beinformation determined on the basis of a plurality of labels predictedby a plurality of diagnosis assistance neural network models. Thediagnosis assistance information may correspond to the plurality oflabels predicted by the plurality of diagnosis assistance neural networkmodels.

In other words, a first diagnosis assistance information may bediagnosis assistance information corresponding to a first labelpredicted through a first diagnosis assistance neural network model.Alternatively, the first diagnosis assistance information may bediagnosis assistance information determined in consideration of a firstlabel predicted through a first diagnosis assistance neural networkmodel and a second label predicted through a second diagnosis assistanceneural network model.

Meanwhile, CAM images obtained from a plurality of diagnosis assistanceneural network models may be output. The CAM images may be output when apredetermined condition is satisfied. For example, in any one of thecase in which a first diagnosis assistance information indicates thatthe patient is abnormal in relation to a first characteristic or thecase in which a second diagnosis assistance information indicates thatthe patient is abnormal in relation to a second characteristic, a CAMimage obtained from a diagnosis assistance neural network model, fromwhich diagnosis assistance information indicating that the patient isabnormal has been output, may be output.

A plurality of diagnosis assistance information and/or CAM images may beprovided to a user. The plurality of diagnosis assistance information orthe like may be provided to the user through an output means of thediagnostic device or client device. The diagnosis assistance informationmay be visually output. This will be described in detail below inSection “5. User interface.”

According to an embodiment of the present invention, diagnosisassistance information corresponding to a diagnosis target image mayinclude level information. The level information may be selected from aplurality of levels. The level information may be determined on thebasis of a plurality of diagnostic information and/or findingsinformation obtained through neural network models. The levelinformation may be determined in consideration of suitabilityinformation or quality information of a diagnosis target image. Thelevel information may be determined in consideration of a class intowhich a diagnosis target image is classified by a plurality of neuralnetwork models. The level information may be determined in considerationof numerical values output from a plurality of neural network models.

For example, diagnosis assistance information obtained corresponding toa diagnosis target image may include any one level information selectedfrom a first level information and a second level information. When atleast one abnormal findings information or abnormal diagnosticinformation is obtained among of diagnostic information obtained througha plurality of neural network models, the first level information may beselected as the level information. When, of diagnostic informationobtained through the neural network models does not include abnormalfindings information or abnormal diagnostic information, the secondlevel information may be selected as the level information.

A first level information may be selected as the level information whenat least one numerical value among numerical values obtained through aneural network model exceeds a reference numerical value, and a secondlevel information may be selected as the level information when all ofthe obtained numerical values are less than a reference numerical value.The first of level information may indicate that strong abnormalinformation is present in a diagnosis target image compared with thesecond of level information.

A third level information may be selected as the level information whenit is determined using image analysis or a neural network model that thequality of a diagnosis target image is a reference quality or lower.Alternatively, diagnosis assistance information may include the thirdlevel information together with the first or second level information.

When diagnosis assistance information includes the first levelinformation, a first user guide may be output through an output means.The first user guide may include matters corresponding to at least oneof abnormal findings information or abnormal diagnostic informationincluded in diagnosis assistance information. For example, the firstuser guide may indicate that a more precise test is required for apatient corresponding to abnormal information included in diagnosisassistance information. For example, the first user guide may indicatethat secondary diagnosis (for example, diagnosis in a separate medicalinstitution or a hospital transfer procedure) is required for thepatient. Alternatively, the first user guide may indicate treatmentrequired for the patient. As a specific example, when abnormalinformation on macular degeneration of the patient is obtained bydiagnosis assistance information, the first user guide may includeinjection prescription and a guide on a hospital transfer procedure (forexample, a list of hospitals to which transfer is possible) related tothe patient.

When diagnosis assistance information includes the second levelinformation, a second user guide may be output through an output means.The second user guide may include future care plans related to thepatient corresponding to the diagnosis assistance information. Forexample, the second user guide may indicate the time of next visit andthe next medical course.

When diagnosis target information includes the third level information,a third user guide may be output through an output means. The third userguide may indicate that a diagnosis target image has to be retaken. Thethird user guide may include information on the quality of the diagnosistarget image. For example, the third user guide may include informationon an artifact present in a diagnosis target image (for example, whetherthe artifact is a bright artifact or a dark artifact, or the degreethereof).

The first to third of level information may be output by an output unitof the client device or diagnostic device. Specifically, the first tothird level information may be output through a user interface whichwill be described below.

4.4 Embodiment 2—Diagnosis Assistance System

A diagnosis assistance system according to an embodiment of the presentinvention may include a fundus image obtaining unit, a first processingunit, a second processing unit, a third processing unit, and adiagnostic information output unit.

According to an embodiment of the present invention, the diagnosisassistance system may include a diagnostic device. The diagnostic devicemay include a fundus image obtaining unit, a first processing unit, asecond processing unit, a third processing unit, and/or a diagnosticinformation output unit. However, the present invention is not limitedthereto, and each unit included in the diagnosis assistance system maybe disposed at a proper position in a training device, a diagnosticdevice, a training diagnosis server, and/or a client device.Hereinafter, for convenience of description, the case in which adiagnostic device of a diagnosis assistance system includes a fundusimage obtaining unit, a first processing unit, a second processing unit,a third processing unit, and a diagnostic information output unit willbe described.

FIG. 28 is a view for describing a diagnosis assistance system accordingto an embodiment of the present invention. Referring to FIG. 28, adiagnosis assistance system may include a diagnostic device, and thediagnostic device may include a fundus image obtaining unit, a firstprocessing unit, a second processing unit, a third processing unit, anda diagnostic information output unit.

According to an embodiment of the present invention, a diagnosisassistance system that assists diagnosis of a plurality of diseases onthe basis of a fundus image may include a fundus image obtaining unitconfigured to obtain a target fundus image which is a basis foracquiring diagnosis assistance information on a patient, a firstprocessing unit configured to, for the target fundus image, obtain afirst result related to a first finding of the patient using a firstneural network model, wherein the first neural network model is trainedon the basis of a first fundus image set, a second processing unitconfigured to, for the target fundus image, obtain a second resultrelated to a second finding of the patient using a second neural networkmodel, wherein the second neural network model is trained on the basisof a second fundus image set which is at least partially different fromthe first fundus image set, a third processing unit configured todetermine, on the basis of the first result and the second result,diagnostic information on the patient, and a diagnostic informationoutput unit configured to provide the determined diagnostic informationto a user. Here, the first finding and the second finding may be usedfor diagnosing different diseases.

The first neural network model may be trained to classify an inputfundus image as any one of a normal label and an abnormal label inrelation to the first finding, and the first processing unit may obtainthe first result by classifying the target fundus image as any one ofthe normal label and the abnormal label using the first neural networkmodel.

The third processing unit may determine whether diagnostic informationaccording to the target fundus image is normal information or abnormalinformation by taking the first result and the second result intoconsideration together.

The third processing unit may determine diagnostic information on thepatient by assigning priority to the abnormal label so that accuracy ofdiagnosis is improved.

When the first label is a normal label related to the first finding, andthe second label is a normal label related to the second finding, thethird processing unit may determine the diagnostic information asnormal. When the first label is not the normal label related to thefirst finding, or the second label is not the normal label related tothe second finding, the third processing unit may determine thediagnostic information as abnormal.

The first finding may be related to an eye disease, and the first resultmay indicate whether the patient is normal in relation to the eyedisease. The second finding may be related to a systemic disease, andthe second result may indicate whether the patient is normal in relationto the systemic disease.

The first finding may be related to a first eye disease, and the firstresult may indicate whether the patient is normal in relation to thefirst eye disease. The second finding may be related to a second eyedisease distinguished from the first eye disease, and the second resultmay indicate whether the patient is normal in relation to the second eyedisease.

The first finding may be a finding for diagnosing a first eye disease,and the first result may indicate whether the patient is normal inrelation to the first eye disease. The second finding may be a findingdistinguished from the first finding for diagnosing the first eyedisease, and the second result may indicate whether the patient isnormal in relation to a second eye disease.

The first neural network model may include a first sub-neural networkmodel and a second sub-neural network model, and the first result may bedetermined by taking a first predicted value predicted by the firstsub-neural network model and a second predicted value predicted by thesecond sub-neural network model into consideration together.

The first processing unit may obtain a CAM related to the first labelthrough the first neural network model, and the diagnostic informationoutput unit may output an image of the CAM.

The diagnostic information output unit may output an image of the CAMwhen the diagnostic information obtained by the third processing unit isabnormal diagnostic information.

The diagnosis assistance system may further include a fourth processingunit configured to obtain quality information on the target fundusimage, and the diagnostic information output unit may output the qualityinformation on the target fundus image obtained by the fourth processingunit.

When it is determined in the fourth processing unit that the qualityinformation on the target fundus image is at a predetermined qualitylevel or lower, the diagnostic information output unit may provideinformation indicating that the quality information on the target fundusimage is at the predetermined quality level or lower together with thedetermined diagnostic information to the user.

5. User Interface

According to an embodiment of the present invention, the above-describedclient device or diagnostic device may have a display unit for providingdiagnosis assistance information to the user. In this case, the displayunit may be provided to facilitate providing of diagnosis assistanceinformation to the user and obtaining of feedback from the user.

As an example of the display unit, a display configured to providevisual information to the user may be provided. In this case, agraphical user interface for visually transferring diagnosis assistanceinformation to the user may be used. For example, in a fundus diagnosisassistance system that obtains diagnosis assistance information on thebasis of a fundus image, a graphical user interface for effectivelydisplaying obtained diagnosis assistance information and helpingunderstanding of the user may be provided.

FIGS. 29 and 30 are views for describing a graphical user interface forproviding diagnostic information to the user according to someembodiments of the present invention. Hereinafter, some embodiments of auser interface that may be used in a fundus diagnosis assistance systemwill be described with reference to FIGS. 29 and 30.

Referring to FIG. 29, a user interface according to an embodiment of thepresent invention may display identification information of a patientcorresponding to a diagnosis target fundus image. The user interface mayinclude a target image identification information display unit 401configured to display identification information of a patient and/orimaging information (for example, the data and time of imaging) of adiagnosis target fundus image.

The user interface according to an embodiment of the present inventionmay include a fundus image display unit 405 configured to display afundus image of the left eye and a fundus image of the right eye of thesame patient. The fundus image display unit 405 may also display a CAMimage.

The user interface according to an embodiment of the present inventionmay include a diagnostic information indicating unit 403 configured toindicate each of the fundus image of the left eye and the fundus imageof the right eye as the image of the left eye or right eye andconfigured to display diagnostic information on each image and adiagnostic information indicator indicating whether the user hasconfirmed the diagnostic information.

Color of the diagnostic information indicator may be determined inconsideration of diagnosis assistance information obtained on the basisof the target fundus image. The diagnostic information indicator may bedisplayed in a first color or a second color according to the diagnosisassistance information. For example, the diagnostic informationindicator may be displayed in red when first to third diagnosisassistance information are obtained from a single target fundus imageand when any one of the of diagnosis assistance information includesabnormal information (that is, indicates that there are abnormalfindings), and the diagnostic information indicator may be displayed ingreen when all of the of diagnosis assistance information includesnormal information (that is, indicates there are not abnormal findings).

The form of the diagnostic information indicator may be determinedaccording to whether the user has confirmed the diagnostic information.The diagnostic information indicator may be displayed in a first form ora second form according to whether the user has confirmed the diagnosticinformation. For example, referring to FIG. 29, a diagnostic informationindicator corresponding to a target fundus image that has been reviewedby the user may be displayed as a filled circle, and a diagnosticinformation indicator corresponding to a target fundus image that hasnot been reviewed by the user yet may be displayed as a half-circle.

The user interface according to an embodiment of the present inventionmay include a diagnostic information indicating unit 407 configured toindicate diagnosis assistance information. The diagnosis assistanceinformation indicating may be disposed at each of the left eye image andthe right eye image. The diagnosis assistance information indicatingunit may indicate a plurality of findings information or diagnosticinformation.

The diagnosis assistance information indicating unit may include atleast one diagnosis assistance information indicator. The diagnosisassistance information indicator may indicate corresponding diagnosisassistance information through a color change.

For example, when, in relation to a diagnosis target fundus image, afirst diagnosis assistance information indicating the presence of theopacity of crystalline lens is obtained through a first diagnosisassistance neural network model, a second diagnosis assistanceinformation indicating the presence of abnormal findings of diabeticretinopathy is obtained through a second diagnosis assistance neuralnetwork model, and a third diagnosis assistance information indicatingthe presence of abnormal findings of the retina is obtained through athird diagnosis assistance neural network model, the diagnosticinformation indicating unit may include first to third diagnosisassistance information indicators configured to respectively indicatethe first diagnosis assistance information, the second diagnosisassistance information, and the third diagnosis assistance information.

As a more specific example, referring to FIG. 29, when, in relation tothe left eye fundus image of the patient, a first diagnosis assistanceinformation indicating that the obtained diagnosis assistanceinformation is abnormal in terms of the opacity of crystalline lens isobtained, a second diagnosis assistance information indicating that theobtained diagnosis assistance information is normal (has no abnormalfindings) in terms of diabetic retinopathy is obtained, and a thirddiagnosis assistance information indicating that the obtained diagnosisassistance information is abnormal (has abnormal findings) in terms ofthe retina is obtained, the diagnostic information indicating unit 407may display a first diagnosis assistance information indicator with afirst color, a second diagnosis assistance information indicator with asecond color, and a third diagnosis assistance information indicatorwith the first color.

The user interface according to an embodiment of the present inventionmay obtain a user comment on a diagnosis target fundus image from theuser. The user interface may include a user comment object 409 and maydisplay a user input window in response to a user selection on the usercomment object. A comment obtained from the user may also be used inupdating a diagnosis assistance neural network model. For example, theuser input window displayed in response to the user's selection on theuser comment object may obtain a user's evaluation on diagnosisassistance information obtained through a neural network, and theobtained user's evaluation may be used in updating a neural networkmodel.

The user interface according to an embodiment of the present inventionmay include a review indicating object 411 configured to display whetherthe user has reviewed each diagnosis target fundus image. The reviewindicating object may receive a user input indicating that the user'sreviewing of each diagnosis target image has been completed, and displaythereof may be changed from a first state to a second state. Referringto FIGS. 29 and 30, upon receiving a user input, the review indicatingobject may be changed from a first state in which a review requestmessage is displayed to a second state indicating that the reviewing hasbeen completed.

A diagnosis target fundus image list 413 may be displayed. In the list,identification information of the patient, the data on which the imagehas been captured, and the indicator 403 of whether the use has reviewedimages of the both eyes may be displayed together.

In the diagnosis target fundus image list 413, a review completionindicator 415 indicating that the corresponding diagnosis target fundusimage has been reviewed may be displayed. The review completionindicator 415 may be displayed when a user selection has been made forreview indicating objects 411 of the both eyes of the correspondingimages.

Referring to FIG. 30, the graphical user interface may include a poorquality warning object 417 indicating that there is an abnormality inthe quality of a diagnosis target fundus image to the user when it isdetermined that there is an abnormality in the quality of the diagnosistarget fundus image. The poor quality warning object 417 may bedisplayed when it is determined that the quality of the diagnosis targetfundus image from the diagnostic unit is below a quality level at whichappropriate diagnosis assistance information may be predicted from adiagnosis assistance neural network model (that is, a reference qualitylevel).

In addition, referring to FIG. 30, the poor quality warning object 417may also be displayed in the diagnosis target fundus image list 413.

According to the present invention, diagnosis assistance information canbe promptly obtained on the basis of a fundus image.

Further, according to the present invention, diagnosis assistanceinformation related to a plurality of diseases can be more accuratelypredicted on the basis of a fundus image.

Further, according to the present invention, obtaining an optimizedneural network model for predicting diagnosis assistance informationrelated to a plurality of diseases can be facilitated.

Advantageous effects of the present invention are not limited to thosementioned above, and other unmentioned advantageous effects should beclearly understood by one of ordinary skill in the art to which thepresent invention pertains from the present specification and theaccompanying drawings.

While the invention has been described above with a few embodiments andthe accompanying drawings, one of ordinary skill in the art may makevarious modifications and changes to the description above. For example,appropriate results can be achieved even if the above-describedtechniques are performed in a different order from that in theabove-described method, and/or the above-described elements such assystems, structures, devices, and circuits are coupled or combined indifferent forms from those in the above-described method or are replacedor substituted with other elements or their equivalents.

Therefore, other implementations, other embodiments, and equivalents tothe claims are also within the scope of the following claims.

What is claimed is:
 1. A diagnosis assistant system for assistingdiagnosis of a plurality of diseases based on an eye image, comprising:an eye image obtaining unit configured to acquire a target eye imagewhich is a basis for acquiring diagnosis assistance information on asubject; a first processing unit configured to, for the target eyeimage, obtain a first result related to a first disease of the subjectusing a first machine learning model; a second processing unitconfigured to, for the target eye image, obtain a second result relatedto a second disease of the subject using a second machine learningmodel, wherein at least part of the first machine learning model isdifferent from the second machine learning model; and a diagnosticinformation output unit configured to provide the diagnosis assistantinformation related to the first result and the second result to a user,wherein the first disease and the second disease are different diseaseseach other, wherein the first disease is related to a first eye diseaseand the first result is used for determining whether the subject isnormal or not regarding the first eye disease, and wherein the seconddisease is related to a second eye disease and the second result is usedfor determining whether the subject is normal or not regarding thesecond eye disease.
 2. The diagnosis assistant system of claim 1,wherein the eye image includes at least one vessel of the eye of thesubject.
 3. The diagnosis assistant system of claim 1, wherein the eyeimage includes a retinal image or a fundus image.
 4. The diagnosisassistant system of claim 1, wherein: the first machine learning modelcomprises a first neural network model, and the second machine learningmodel comprises a second neural network model.
 5. The diagnosisassistant system of claim 1, wherein: the first machine learning modelis trained to classify an input eye image into one of normal label andan abnormal label regarding the first disease, and the first processingunit obtains the first result by classifying the target eye image intoone of the normal label or the abnormal label.
 6. The diagnosisassistant system of claim 1, wherein the first processing unit obtains afirst map related to the first result via the first machine learningmodel and the diagnostic information output unit outputs an image of thefirst map.
 7. The diagnosis assistant system of claim 1, furthercomprising: a third processing unit configured to obtain a qualityinformation of the target eye image, and wherein the diagnosticinformation output unit outputs the quality information of the targetfundus image obtained by the third processing unit.
 8. A diagnosisassistant system for assisting diagnosis of a plurality of diseasesbased on an eye image, comprising: an eye image obtaining unitconfigured to acquire a target eye image which is a basis for acquiringdiagnosis assistance information on a subject; a first processing unitconfigured to, for the target eye image, obtain a first result relatedto a first disease of the subject using a first machine learning model;a second processing unit configured to, for the target eye image, obtaina second result related to a second disease of the subject using asecond machine learning model, wherein at least part of the firstmachine learning model is different from the second machine learningmodel; and a diagnostic information output unit configured to providethe diagnosis assistant information related to the first result and thesecond result to a user, wherein the first disease and the seconddisease are different diseases each other, wherein the first disease isrelated to an eye disease and the first result is used for determiningwhether the subject is normal or not regarding the eye disease, andwherein the second disease is related to a systemic disease and thesecond result is used for determining whether the subject is normal ornot regarding the systemic disease.
 9. The diagnosis assistant system ofclaim 8, wherein the systemic disease comprises at least one ofhypertension, diabetes, Alzheimer's disease, cytomegalovirus disease,stroke, arteriosclerosis and cardiovascular disease.
 10. The diagnosisassistant system of claim 8, further comprising: a third processing unitconfigured to obtain a quality information of the target eye image, andwherein the diagnostic information output unit outputs the qualityinformation of the target fundus image obtained by the third processingunit.
 11. A diagnosis assistant system for assisting diagnosis of aplurality of diseases based on an eye image, comprising: an eye imageobtaining unit configured to acquire a target eye image which is a basisfor acquiring diagnosis assistance information on a subject; a dataprocessing unit configured to extract features of the target eye image;a first processing unit configured to, for the features of the targeteye image, obtain a first result related to a first disease of thesubject using a first machine learning model; a second processing unitconfigured to, for the features of the target eye image, obtain a secondresult related to a second disease of the subject using a second machinelearning model, wherein the first machine learning model is differentfrom the second machine learning; and a diagnostic information outputunit configured to provide the diagnosis assistant information relatedto the first result and the second result to a user, wherein the firstdisease and the second disease are different diseases each other,wherein the first disease is related to a first eye disease and thefirst result is used for determining whether the subject is normal ornot regarding the first eye disease, and wherein the second disease isrelated to a second eye disease and the second result is used fordetermining whether the subject is normal or not regarding the secondeye disease.
 12. The diagnosis assistant system of claim 11, furthercomprising: a third processing unit configured to obtain a qualityinformation of the target eye image, and wherein the diagnosticinformation output unit outputs the quality information of the targetfundus image obtained by the third processing unit.
 13. A diagnosisassistant system for assisting diagnosis of a plurality of diseasesbased on an eye image, comprising: an eye image obtaining unitconfigured to acquire a target eye image which is a basis for acquiringdiagnosis assistance information on a subject; a data processing unitconfigured to extract features of the target eye image; a firstprocessing unit configured to, for the features of the target eye image,obtain a first result related to a first disease of the subject using afirst machine learning model; a second processing unit configured to,for the features of the target eye image, obtain a second result relatedto a second disease of the subject using a second machine learningmodel, wherein the first machine learning model is different from thesecond machine learning; and a diagnostic information output unitconfigured to provide the diagnosis assistant information related to thefirst result and the second result to a user, wherein the first diseaseand the second disease are different diseases each other, wherein thefirst disease is related to an eye disease and the first result is usedfor determining whether the subject is normal or not regarding the eyedisease, and wherein the second disease is related to a systemic diseaseand the second result is used for determining whether the subject isnormal or not regarding the systemic disease.
 14. The diagnosisassistant system of claim 13, further comprising: a third processingunit configured to obtain a quality information of the target eye image,and wherein the diagnostic information output unit outputs the qualityinformation of the target fundus image obtained by the third processingunit.
 15. A method for providing diagnosis assistance information forassisting diagnosis of a plurality of diseases based on an eye image,the method executed on one or more processors, the method comprising:acquiring a target eye image which is a basis for acquiring thediagnosis assistance information on a subject; obtaining, for the targeteye image, a first result related to a first disease of the subjectusing a first machine learning model; obtaining, for the target eyeimage, a second result related to a second disease of the subject usinga second machine learning model, wherein at least part of the firstmachine learning model is different from the second machine learningmodel; and providing the diagnosis assistant information related to thefirst result and the second result to a user, wherein the first diseaseand the second disease are different diseases each other, wherein thefirst disease is related to a first eye disease and the first result isused for determining whether the subject is normal or not regarding thefirst eye disease, and wherein the second disease is related to a secondeye disease and the second result is used for determining whether thesubject is normal or not regarding the second eye disease.
 16. Anon-transitory computer-readable recording medium having recordedthereon a program for performing the method of claim
 15. 17. A methodfor providing diagnosis assistance information for assisting diagnosisof a plurality of diseases based on an eye image, the method executed onone or more processors, the method comprising: acquiring a target eyeimage which is a basis for acquiring the diagnosis assistanceinformation on a subject; obtaining, for the target eye image, a firstresult related to a first disease of the subject using a first machinelearning model; obtaining, for the target eye image, a second resultrelated to a second disease of the subject using a second machinelearning model, wherein at least part of the first machine learningmodel is different from the second machine learning model; and providingthe diagnosis assistant information related to the first result and thesecond result to a user, wherein the first disease and the seconddisease are different diseases each other, wherein the first disease isrelated to an eye disease and the first result is used for determiningwhether the subject is normal or not regarding the eye disease, andwherein the second disease is related to a systemic disease and thesecond result is used for determining whether the subject is normal ornot regarding the systemic disease.
 18. A non-transitorycomputer-readable recording medium having recorded thereon a program forperforming the method of claim
 17. 19. A method for providing diagnosisassistance information for assisting diagnosis of a plurality ofdiseases based on an eye image, the method executed on one or moreprocessors, the method comprising: acquiring a target eye image which isa basis for acquiring the diagnosis assistance information on a subject;extracting features of the target eye image; obtaining, for the featuresof the target eye image, a first result related to a first disease ofthe subject using a first machine learning model; obtaining, for thefeatures of the target eye image, a second result related to a seconddisease of the subject using a second machine learning model, whereinthe first machine learning model is different from the second machinelearning; and providing the diagnosis assistant information related tothe first result and the second result to a user, wherein the firstdisease and the second disease are different diseases each other,wherein the first disease is related to a first eye disease and thefirst result is used for determining whether the subject is normal ornot regarding the first eye disease, and wherein the second disease isrelated to a second eye disease and the second result is used fordetermining whether the subject is normal or not regarding the secondeye disease.
 20. A non-transitory computer-readable recording mediumhaving recorded thereon a program for performing the method of claim 19.21. A method for providing diagnosis assistance information forassisting diagnosis of a plurality of diseases based on an eye image,the method executed on one or more processors, the method comprising:acquiring a target eye image which is a basis for acquiring thediagnosis assistance information on a subject; extracting features ofthe target eye image; obtaining, for the features of the target eyeimage, a first result related to a first disease of the subject using afirst machine learning model; obtaining, for the features of the targeteye image, a second result related to a second disease of the subjectusing a second machine learning model, wherein the first machinelearning model is different from the second machine learning; andproviding the diagnosis assistant information related to the firstresult and the second result to a user, wherein the first disease andthe second disease are different diseases each other, wherein the firstdisease is related to an eye disease and the first result is used fordetermining whether the subject is normal or not regarding the eyedisease, and wherein the second disease is related to a systemic diseaseand the second result is used for determining whether the subject isnormal or not regarding the systemic disease.
 22. A non-transitorycomputer-readable recording medium having recorded thereon a program forperforming the method of claim 21.